Advanced Trackers Research Articles

Temporal relation transformer for robust visual tracking with dual-memory learning

Recently, transformer trackers mostly associate multiple reference images with the search area to adapt to the changing appearance of the target. However, they ignore the learned cross-relations between the target and surrounding, leading to difficulties in building coherent contextual models for specific target instances. This paper presents a Temporal Relation Transformer Tracker (TRTT) for robust visual tracking, providing a concise approach to modeling temporal relations by dual target memory learning. Specifically, a temporal relation transformer network generates paired memories based on static and dynamic templates, which are reinforced interactively. The memory contains implicit relation hints that capture the relations between the tracked object and its immediate surroundings. More importantly, to ensure consistency of target instance identities between frames, the relation hints from previous frames are transferred to the current frame for merging temporal contextual attention. Our method also incorporates mechanisms for reusing favorable cross-relations and instance-specific features, thereby overcoming background interference in complex spatio-temporal interactions through a sequential constraint. Furthermore, we design a memory token sparsification method that leverages the key points of the target to eliminate interferences and optimize attention calculations. Extensive experiments demonstrate that our method surpasses advanced trackers on 8 challenging benchmarks while maintaining real-time running speed.

FQTrack:Object Tracking Method Based on a Feature-Enhanced Memory Network and Memory Quality Selection Mechanism

Visual object tracking technology is widely used in intelligent security, automatic driving and other fields, and also plays an important role in frontier fields such as human–computer interactions and virtual reality. The memory network improves the stability and accuracy of tracking by using historical frame information to assist in the positioning of the current frame in object tracking. However, the memory network is still insufficient in feature mining and the accuracy and robustness of the model may be reduced when using noisy observation samples to update it. In view of the above problems, we propose a new tracking framework, which uses the attention mechanism to establish a feature-enhanced memory network and combines cross-attention to aggregate the spatial and temporal context information of the target. The former introduces spatio-temporal adaptive attention and cross-spatial attention, embeds spatial location information into channels, realizes multi-scale feature fusion, dynamically emphasizes target location information, and obtains richer feature maps. The latter guides the tracker to focus on the area with the largest amount of information in the current frame to better distinguish the foreground and background. In addition, through the memory quality selection mechanism, the accuracy and richness of the feature samples are improved, thereby enhancing the adaptability and discrimination ability of the tracking model. Experiments on benchmark test sets such as OTB2015, TrackingNet, GOT-10k, LaSOT and UAV 123 show that this method achieves comparable performance with advanced trackers.

MASNet: mixed attention Siamese network for visual object tracking

Most Siamese-based trackers adopt correlation operation to perform similarity matching on feature fusion of template branch and search branch. However, the correlation operation directly uses the template feature to slide the window on the search area feature, and it is difficult to distinguish the target and background information when encountering similar target interference and background clutter, which can easily lead to tracking failure. In this paper, a mixed attention Siamese Network (MASNet) is proposed, which neatly integrates the modelling of global information and mutual information into the Siamese tracking framework. Specifically, through the learnable attention module, the local and global dependencies are adaptively integrated, and the rich context information is captured to improve the discrimination of the network for similar targets. After that, the correlation of the context information of the two branches is aggregated and adaptively encoded into the two branches, which makes up for the shortcomings of single correlation operation. In addition, the framework adopts a three-layer feature fusion method, which enables the tracker to better adapt to object deformation and similar object interference. Extensive experiments on OTB100, UAV123, GOT-10k and LaSOT show that the proposed tracking method achieves comparable performance to the advanced trackers.

EMAT: Efficient feature fusion network for visual tracking via optimized multi-head attention

The tracking methods based on Transformer have shown great potential in visual tracking and achieved significant tracking performance. The traditional transformer based feature fusion network divides a whole feature map into multiple image patches as its inputs, and then directly processes them in parallel, which will occupy a lot of computing resources and affect the computing efficiency of multi-head attention. In this paper, we design a novel feature fusion network with optimized multi-head attention in encoder and decoder architecture based on Transformer. The designed feature fusion network preprocess the input features and change the calculations of multi-head attention by using both the efficient multi-head self-attention module and efficient multi-head spatial reduction attention module. The two modules can reduce the influence of irrelevant background information, enhance the representation ability of template features and search region features, and greatly reduce the computational complexity. We propose a novel Transformer tracking method (named EMAT) based on the designed feature fusion network. The proposed EMAT is evaluated on seven challenging tracking benchmarks to demonstrate its superiority, including LaSOT, GOT-10k, TrackingNet, UAV123, VOT2018, NfS and VOT-RGBT2019. The proposed tracker achieves well tracking performance, and obtains precision score of 89.0% on UAV123, AUC score of 64.6% on LaSOT, EAO score of 34.8% on VOT-RGBT2019, which outperforms most advanced trackers. EMAT runs at a real-time speed of about 35 FPS during tracking.

Toward Class-Agnostic Tracking Using Feature Decorrelation in Point Clouds.

Single object tracking in point clouds has been attracting more and more attention owing to the presence of LiDAR sensors in 3D vision. However, existing methods based on deep neural networks mainly focus on training different models for different categories, which makes them unable to perform well in real-world applications when encountering classes unseen during the training phase. In this work, we investigate a more challenging task in LiDAR point clouds, namely class-agnostic tracking, where a general model is supposed to be learned to handle targets of both observed and unseen categories. In particular, we first investigate the class-agnostic performance of state-of-the-art trackers by exposing the unseen categories to them during testing. It is found that as the distribution shifts from observed to unseen classes, how to constrain the fused features between the template and the search region to maintain generalization is a key factor in class-agnostic tracking. Therefore, we propose a feature decorrelation method to address this problem, which eliminates the spurious correlations of the fused features through a set of learned weights, and further makes the search region consistent among foreground points and distinctive between foreground and background points. Experiments on KITTI and NuScenes demonstrate that the proposed method can achieve considerable improvements by benchmarking against the advanced trackers P2B and BAT, especially when tracking unseen objects.

Multi-Task Probabilistic Regression With Overlap Maximization for Visual Tracking

Recent researches made a breakthrough in visual tracking accuracy. Many trackers benefit from the object state representations and network loss functions, which mine the output space and improve the power of supervision, respectively. Probabilistic regression method models the noises and uncertainties in the annotations. However, advanced trackers with probabilistic regression are not studied sufficiently in the aspect of supervision and the aspect of robustness of evaluation maximization. In this paper, an overlap maximization network in the manner of probabilistic regression is proposed to improve the learning ability of the network and the discriminative ability in the evaluation maximization. Firstly, the probabilistic regression is extended with the intersection over union (IoU) evaluation, which is normalized as a probability density in the regression space. Secondly, the classification probability is added as a branch of the iterative evaluation module to improve the ability of distinguishing objects in the evaluation maximization. Moreover, the two branches are constructed into a joint probabilistic regression task of IoU evaluation, which makes the network learn from two types of ground truth and provide a consistent result with multi-branch outputs. For feature interpretation, the strip pooling network and the space-time memory network are introduced to encode long-range context and provide dynamic features, respectively. Compared to the state-of-the-art probabilistic regression trackers and other advanced trackers, the experiments show that the proposed tracker achieves outstanding performance across the six datasets, including GOT-10k, LaSOT, TrackingNet, UAV123, OTB-100 and VOT2018.

IoUformer: Pseudo-IoU prediction with transformer for visual tracking

Siamese tracking has witnessed tremendous progress in tracking paradigm. However, its default box estimation pipeline still faces a crucial inconsistency issue, namely, the bounding box decided by its classification score is not always best overlapped with the ground truth, thus harming performance. To this end, we explore a novel simple tracking paradigm based on the intersection over union (IoU) value prediction. To first bypass this inconsistency issue, we propose a concise target state predictor termed IoUformer, which instead of default box estimation pipeline directly predicts the IoU values related to tracking performance metrics. In detail, it extends the long-range dependency modeling ability of transformer to jointly grasp target-aware interactions between target template and search region, and search sub-region interactions, thus neatly unifying global semantic interaction and target state prediction. Thanks to this joint strength, IoUformer can predict reliable IoU values near-linear with the ground truth, which paves a safe way for our new IoU-based siamese tracking paradigm. Since it is non-trivial to explore this paradigm with pleased efficacy and portability, we offer the respective network components and two alternative localization ways. Experimental results show that our IoUformer-based tracker achieves promising results with less training data. For its applicability, it still serves as a refinement module to consistently boost existing advanced trackers.

Sparse mixed attention aggregation network for multimodal images fusion tracking

Recent years have witnessed the exciting performance of trackers based on Transformer. However, they usually separate the process of information extraction and integration, weakening the information interaction between the target and search region. In addition, they depend on traditional Transformer to model the long range dependency, which leads to a lack of focus on the primary information needed by high-accuracy trackers. In this paper, a sparse mixed attention aggregation model is proposed for robust tracking based on visible and thermal infrared images. To be specific, a backbone network composed of sparse mixed attention is designed to achieve information extraction and integration. This is helpful to obtain specific discriminative feature information and enhance their communication. To give full play to the complementary visible and thermal information, a confidence aware aggregation network is designed, which can learn the reliable confidence of visible and thermal branches. Finally, a corner-based localization head is introduced to estimate the target state. Extensive experiments on three large-scale multimodal tracking benchmarks demonstrate the superior tracking ability of the proposed tracker over other advanced trackers.

Object tracking based on siamese network with 3D attention and multiple graph attention

Currently, the object tracking algorithm based on the siamese network is the most popular research direction of object tracking. However, most siamese network trackers are unable to update the template, resulting in hardly dealing with the fuzzy target well, which requires the trackers to extract the target features in the template more accurately. Moreover, most of the siamese network trackers compute the similarity between the target template and the search region in a global matching way. Thus they cannot well handle the rapid change of the target. To address the above limitations, we propose an object tracking algorithm based on 3D attention SimAM and multiple graph attention. We combine siamese networks with 3D attention to improve the feature extraction capability of the network. We frame the target template and search region to improve feature-matching accuracy by introducing multiple graph attention mechanisms. We test our model against three authoritative benchmarks, GOT10K, UAV123 and OTB100, and compare it with advanced trackers, and the results show that our trackers achieve better results.

STFTrack: Spatio-Temporal-Focused Siamese Network for Infrared UAV Tracking

The rapid popularity of UAVs has encouraged the development of Anti-UAV technology. Infrared-detector-based visual tracking for UAVs provides an encouraging solution for Anti-UAVs. However, it still faces the problem of tracking instability caused by environmental thermal crossover and similar distractors. To address these issues, we propose a spatio-temporal-focused Siamese network for infrared UAV tracking, called STFTrack. This method employs a two-level target focusing strategy from global to local. First, a feature pyramid-based Siamese backbone is constructed to enhance the feature expression of infrared UAVs through cross-scale feature fusion. By combining template and motion features, we guide prior anchor boxes towards the suspicious region to enable adaptive search region selection, thus effectively suppressing background interference and generating high-quality candidates. Furthermore, we propose an instance-discriminative RCNN based on metric learning to focus on the target UAV among candidates. By measuring calculating the feature distance between the candidates and the template, it assists in discriminating the optimal target from the candidates, thus improving the discrimination of the proposed method to infrared UAV. Extensive experiments on the Anti-UAV dataset demonstrate that the proposed method achieves outstanding performance for infrared tracking, with 91.2% precision, 66.6% success rate, and 67.7% average overlap accuracy, and it exceeded the baseline algorithm by 2.3%, 2.7%, and 3.5%, respectively. The attribute-based evaluation demonstrates that the proposed method achieves robust tracking effects on challenging scenes such as fast motion, thermal crossover, and similar distractors. Evaluation on the LSOTB-TIR dataset shows that the proposed method reaches a precision of 77.2% and a success rate of 63.4%, outperforming other advanced trackers.

Transformer tracking with multi-scale dual-attention

Transformer-based trackers greatly improve tracking success rate and precision rate. Attention mechanism in Transformer can fully explore the context information across successive frames. Nevertheless, it ignores the equally important local information and structured spatial information. And irrelevant regions may also affect the template features and search region features. In this work, a multi-scale feature fusion network is designed with box attention and instance attention in Encoder–Decoder architecture based on Transformer. After extracting features, the local information and structured spatial information is learnt by multi-scale box attention, and the global context information is explored by instance attention. Box attention samples grid features from the region of interest. Therefore, it effectively focuses on the region of interest (ROI) and avoids the influence of irrelevant regions in feature extraction. At the same time, instance attention can also pay attention to the context information across successive frames, and avoid falling into local optimum. The long-range feature dependencies are learned in this stage. Extensive experiments are conducted on six challenging tracking datasets to demonstrate the superiority of the proposed tracker MDTT, including UAV123, GOT-10k, LaSOT, VOT2018, TrackingNet, and NfS. In particular, the proposed tracker achieves AUC score of 64.7 % on LaSOT, 78.1 % on TrackingNet and precision score of 89.2 % on UAV123, which outperforms the baseline and most recent advanced trackers.

SiamST: Siamese network with spatio-temporal awareness for object tracking

Recently, Siamese trackers have achieved remarkable tracking performance. However, challenges such as accurate feature representation of targets with different spatial regions and the utilization of diverse target temporal states still need to be addressed. Here, we proposed a Siamese network with spatio-temporal awareness called SiamST. The standard square convolutional kernels and the single feature matching operation hardly represent the targets with different shapes accurately. Therefore, we designed a refined region fusion module that combines multiple convolutional kernels to fit targets with different aspect ratios. Furthermore, we proposed a multi-granularity matching module to obtain more robust feature matching results by combining fine-grained and coarse-grained matching results. However, most existing Siamese trackers do not adequately employ target temporal states. They usually only update the templates, which automatically causes motion information loss. Therefore, we built dynamic templates by screening high-quality samples to describe the target appearance changes accurately. In addition, we designed a trend guidance module to adjust the location prior constraint appropriately to match the tracking results to the target's motion trajectory. Extensive experimental results on eight tracking benchmarks demonstrate the competitive performance of SiamST compared to many advanced trackers.

TMTNet: A Transformer-Based Multimodality Information Transfer Network for Hyperspectral Object Tracking

Hyperspectral video with spatial and spectral information has great potential to improve object tracking performance. However, the limited hyperspectral training samples hinder the development of hyperspectral object tracking. Since hyperspectral data has multiple bands, from which any three bands can be extracted to form pseudocolor images, we propose a Transformer-based multimodality information transfer network (TMTNet), aiming to improve the tracking performance by efficiently transferring the information of multimodality data composed of RGB and hyperspectral in the hyperspectral tracking process. The multimodality information needed to be transferred mainly includes the RGB and hyperspectral multimodality fusion information and the RGB modality information. Specifically, we construct two subnetworks to transfer the multimodality fusion information and the robust RGB visual information, respectively. Among them, the multimodality fusion information transfer subnetwork is designed based on the dual Siamese branch structure. The subnetwork employs the pretrained RGB tracking model as the RGB branch to guide the training of the hyperspectral branch with little training samples. The RGB modality information transfer subnetwork is designed based on a pretrained RGB tracking model with good performance to improve the tracking network’s generalization and accuracy in unknown complex scenes. In addition, we design an information interaction module based on Transformer in the multimodality fusion information transfer subnetwork. The module can fuse multimodality information by capturing the potential interaction between different modalities. We also add a spatial optimization module to TMTNet, which further optimizes the object position predicted by the subject network by fully retaining and utilizing detailed spatial information. Experimental results on the only available hyperspectral tracking benchmark dataset show that the proposed TMTNet tracker outperforms the advanced trackers, demonstrating the effectiveness of this method.

Adaptive temporal feature modeling for visual tracking via cross-channel learning

Convolution Neural Networks (CNNs) based trackers achieve excellent tracking performance on tracking accuracy and speed. Feature extraction from the target template and search regions is a key part in visual tracking. Recently, existing feature subnetworks combine CNNs with channel attention for feature extraction. However, some existing feature subnetworks do not make the best of the target location dependencies, which result the target location dependency information lost in extracted target features. In this work, we design a novel feature extraction subnetwork with local temporal adaptive modules to obtain location sensitive importance maps, which effectively capture the diverse motion information and highlight the target location information. The target location dependency information is fully utilized to obtain more accurate target location information of the target template and search region in feature extraction subnetwork. The feature extraction subnetwork also fully exploits the local temporal semantics. Furthermore, we learn an interactive module in the template branch, which further captures the non-linear cross-channel interaction and channel-wise dependencies by combining every channel and its k neighbors. The template branch further utilizes cross-channel interactions for capturing the channel dependencies. The interactive module only increases a little extra computational burden. Comparing with other attention modules for visual tracking, this interactive module is lightweight. We propose a novel tracking framework, which mainly includes the designed feature extraction subnetwork and the interactive learning module. We evaluate the proposed tracker on GOT-10k, UAV123, DTB70, NFS, OTB-100, VOT2018, LaSOT and VOT-RGBT2019 benchmarks against advanced trackers, achieving leading performance with 60 FPS tracking speed.

Anchor-Free Tracker Based on Space-Time Memory Network

In the visual object tracking task, the existing trackers cannot well solve the appearance deformation, occlusion, and similar object interference, etc. To address these problems, this article proposes a new Anchor-free Tracker based on Space-time Memory Network (ATSMN). In this work, we innovatively use the space-time memory network, memory feature fusion network, and transformer feature cross fusion network. Through the synergy of above-mentioned innovations, tracker can make full use of temporal context information in the memory frames related to the object and better adapt to the appearance change of the object, which can obtain accurate classification and regression results. Extensive experimental results on challenging benchmarks show that ATSMN can achieve the SOTA level tracking performance compared with other advanced trackers.

WebUAV-3M: A Benchmark for Unveiling the Power of Million-Scale Deep UAV Tracking.

Unmanned aerial vehicle (UAV) tracking is of great significance for a wide range of applications, such as delivery and agriculture. Previous benchmarks in this area mainly focused on small-scale tracking problems while ignoring the amounts of data, types of data modalities, diversities of target categories and scenarios, and evaluation protocols involved, greatly hiding the massive power of deep UAV tracking. In this article, we propose WebUAV-3M, the largest public UAV tracking benchmark to date, to facilitate both the development and evaluation of deep UAV trackers. WebUAV-3M contains over 3.3 million frames across 4,500 videos and offers 223 highly diverse target categories. Each video is densely annotated with bounding boxes by an efficient and scalable semi-automatic target annotation (SATA) pipeline. Importantly, to take advantage of the complementary superiority of language and audio, we enrich WebUAV-3M by innovatively providing both natural language specifications and audio descriptions. We believe that such additions will greatly boost future research in terms of exploring language features and audio cues for multi-modal UAV tracking. In addition, a fine-grained UAV tracking-under-scenario constraint (UTUSC) evaluation protocol and seven challenging scenario subtest sets are constructed to enable the community to develop, adapt and evaluate various types of advanced trackers. We provide extensive evaluations and detailed analyses of 43 representative trackers and envision future research directions in the field of deep UAV tracking and beyond. The dataset, toolkits, and baseline results are available at https://github.com/983632847/WebUAV-3M.

Context‐aware Siamese network for object tracking

At present, temporal and spatial contexts are widely used to improve the adaptability of a tracker. However, most existing methods usually focus on one aspect of the temporal or spatial context and rarely exploit them simultaneously. In this paper, a context-aware Siamese Network (CSNet) is proposed, which skilfully integrates the modelling of temporal and spatial context into the Siamese tracking framework. Specifically, CSNet consists of a context-based channel attention module and a context-based cross-attention module. The former aggregates spatial context information from different channels and dynamically emphasizes target features, which makes it easier for the tracker to distinguish the target from the background. The latter propagates the temporal context from the previous frames to the current frame to establish the part-level relationship between the search region and the historical target state, which enables the tracker better adapt to the target deformation. In addition, to further mine context information, the CSNet is equipped with a state-aware strategy to control the contribution of different context information in tracking. Extensive experiments on OTB2015, UAV123, GOT-10k, LaSOT, and TrackingNet show that the proposed tracking method achieves comparable performance to the advanced trackers.

Uncertain motion tracking via target-objectness proposal and memory validation

Traditional correlation filter trackers only rely on a search window to localize the object in each frame. However, such approaches are prone to fail in case of e.g.uncertain motion or distractor, where a simple motion model alone is insufficient for robust tracking. To address this problem, we propose a novel tracking architecture that can utilize target-objectness proposals (TOP) effectively. Firstly, we reformulate the Markov Chains Monte Carlo method as a region proposal mechanism from a new perspective. It integrates learning features into the feature space sampling strategy for global object candidates boxes generator. Secondly, to improve the quality of candidates boxes, we constructe objectness labels to guide the sampling process. With the help of objectness information, this interaction of spatial constraints increases the effectiveness of region proposals. In the end, a unified tracking framework is designed to enable sampling and regression strategies to exploit and complement each other to cope with uncertain motion tracking. The proposed TOP tracker performs favorably against advanced trackers, especially in uncertain motion scenarios, on four benchmark datasets including OTB-2013, OTB-2015, Temple Color-128 and UAV-123.

A Unified Multi-Scenario Attacking Network for Visual Object Tracking

Existing methods of adversarial attacks successfully generate adversarial examples to confuse Deep Neural Networks (DNNs) of image classification and object detection, resulting in wrong predictions. However, these methods are difficult to attack models of video object tracking, because the tracking algorithms could handle sequential information across video frames and the categories of targets tracked are normally unknown in advance. In this paper, we propose a Unified and Effective Network, named UEN, to attack visual object tracking models. There are several appealing characteristics of UEN: (1) UEN could produce various invisible adversarial perturbations according to different attack settings by using only one simple end-to-end network with three ingenious loss function; (2) UEN could generate general visible adversarial patch patterns to attack the advanced trackers in the real-world; (3) Extensive experiments show that UEN is able to attack many state-of-the-art trackers effectively (e.g. SiamRPN-based networks and DiMP) on popular tracking datasets including OTB100, UAV123, and GOT10K, making online real-time attacks possible. The attack results outperform the introduced baseline in terms of attacking ability and attacking efficiency.

Adaptive feature fusion for visual object tracking

Recent advanced trackers, consisting of discriminative classification component and dedicated bounding box estimation, have achieved improved performance in the visual tracking community. The most essential factor for the development is the utilization of different Convolutional Neural Networks (CNNs), which significantly improves the model capacity via offline trained deep feature representations. Though powerful deep structures emphasize more semantic appearance through high dimensional latent variables, how to achieve effective feature adaptation in the online tracking stage has not been sufficiently considered yet. To this end, we argue the necessity of exploring hierarchical and complementary appearance descriptors from different convolutional layers to achieve online tracking adaptation. Therefore, in this paper, we propose an adaptive feature fusion mechanism, which can balance the detection granularities from shallow to deep convolutional layers. To be specific, the correlation between template and instance is employed to generate adaptive weights to achieve advanced saliency and discrimination. In addition, considering temporal appearance variation, the projection matrix for the multi-channel inputs is jointly updated with the correlation classifier to further enhance the robustness. The experimental results on four recent benchmarks, i.e., OTB-2015, VOT2018, LaSOT and TrackingNet, demonstrate the effectiveness and robustness of the proposed method, with superior performance compared to the state-of-the-art approaches.