Motion sketch based crowd video retrieval

Abstract

Crowd video retrieval with desired motion flow segmentation is an important problem in surveillance video management, e.g., video indexing and browsing, especially in the age of big data. In this paper, we address this issue from the motion-level perspective by using hand-drawn sketches as queries. Motion sketch based crowd video retrieval naturally suffers from challenges in crowd motion representation and similarity measurement. To tackle them, we propose to (1) leverage the motion structure coding algorithm for motion-level video indexing and hand-drawn sketch representation and (2) exploit distance metric fusion strategy incorporated with Ranking SVM for measuring the relevant degree between a sketch query and crowd videos. Specifically, for video indexing, motion decomposition is utilized to separate sub-motion vector fields with typical patterns from a set of optical flows. Then, the motion-level descriptors of the vector fields are computed and stored in an index database. To represent motion sketches, we propose a mechanism by vectorizing the sketches followed by motion structure coding. In the retrieval stage, we first compute the pairwise distance with different metrics between a new sketch query and crowd videos, and then stack them into a feature vector as the input of the Ranking SVM algorithm. Finally, we use the learned retrieval model to predict the ranking score of each crowd video in the database. Experimental results on the publicly available crowd datasets show the robustness and effectiveness of the proposed sketch based crowd video retrieval system.