Prototype-guided Instance matching for multiple pedestrian tracking

Abstract

Multiple pedestrian tracking is a critical yet challenging to understand human behavior. Although existing tracking methods have achieved great advances through adopting the tracking-by-detection paradigm, they give insufficient consideration on how to improve the instance matching capability in the cluttered and crowded environment. Motivated by our observations on tracking persons that the similarity of the same target is larger than that of other persons along sequence frames, in this work, we propose a novel Prototype-guided Instance Matching (PIM) method to boost the performance of multiple pedestrian tracking task. The whole PIM framework mainly consists of a detector net, an appearance embedding module, and a self-adaptive prototype dictionary. Given the current frame as the input sample, we first adopt a classic detector to select a set of positive candidates, and their corresponding multi-scale convolutional features are then fed into the appearance embedding module for obtaining the robust instance-wise feature representation. We perform the prototype-guided instance matching optimization between the learned instance-wise feature embeddings and the prototype dictionary, where its aim is to enlarge the separability of different instances and make the distance between the same instances more compact. Here object detection and appearance embedding are jointly optimized in an end-to-end fashion. Furthermore, we maintain a self-adaptive prototype dictionary to store instance-wise prototypes, which can be adaptively updated to adapt the appearance variations of all tracked targets over time. The instance matching and prototype dictionary update are modeled into an interactive optimization process so that they can be gradually improved for each other. Comprehensive evaluations on three benchmark datasets (i.e., MOT16, MOT17, and MOT20) demonstrate the effectiveness of our proposed PIM when compared with the state-of-the-art methods for the multiple pedestrian tracking task.