A spatial–temporal contexts network for object tracking

Abstract

Although there have been significant advancements and developments in visual object tracking in recent years, most trackers have failed to adapt to the deterioration of object appearance in complex scenes. Typically, they utilize only spatial information or simple temporal networks. The fusion of spatial and temporal contexts among consecutive frames can hypothetically capture historical information to boost tracking performance but inevitably pollutes the model with noisy samples. To this end, we proposed a novel end-to-end ConvLSTM-based tracking framework called STCTrack, which uses spatial and temporal information from each frame and adapts to noisy samples. Specifically, a multilayer residual ConvLSTM-based spatial–temporal context network (STCN) was proposed in STCTrack to retain the target’s past information and consequently guide the tracker to focus on the most informative regions of the current frame. Furthermore, a multi-similarity map fusion model was proposed to calculate the pixel-level similarity map, allowing STCTrack to adaptively retrieve historical target information from different times and be resilient to partial occlusions and nonrigid deformations. Extensive empirical studies were conducted on benchmarks, including OTB2015, GOT-10K, TrackingNet, LaSOT, UAV123, and VOT2018. The empirical results suggest that the proposed STCTrack achieves state-of-the-art performance compared with existing schemes. The code and models used in this study are publicly available at www.github.com/Kevoen/STCTrack to encourage further research on this topic.