Semantic video fingerprinting and retrieval using face information

Abstract

The management of large video databases, especially those containing motion picture and television data, is a major contemporary challenge. A very significant tool for this management is the ability to retrieve those segments that are perceptually similar to a query segment. Another similar but equally important task is determining if a query segment is a (possibly modified) copy of part of a video in the database. The basic way to perform these two tasks is to characterize each video segment with a unique representation called a signature. Using semantic information for the construction of the signatures is a good way to ensure robustness in retrieval and fingerprinting. Here a ubiquitous semantic feature, namely the existence and identity of human faces, will be used to construct the signature. A fast algorithm has been developed to quickly and robustly perform these two tasks on very large video databases. The prerequisite face recognition was performed by a commercial system. Having verified the basic efficacy of our algorithm on a database of real video from motion pictures and television series, we then proceed to further explore its performance in an artificial digital video database, which was created using a probabilistic model of the video creation process. This enabled us to explore variations in performance based on parameters that were impossible to control in a real video database. Furthermore, the suitability of the proposed approach for very large databases was tested using (artificial) data corresponding to hundreds or thousands of hours of video.