Integrated semantic-syntactic video event modeling for search and retrieval

Abstract

This paper proposes to integrate text-based database models (commonly used in the database and information retrieval literature) and low-level video features, such as object-based motion features (commonly used in image processing and computer vision), within a single framework to describe video events for fast and effective browsing and retrieval using a mixture of textual and low-level descriptors. The instantiations of the model are expressed as graphs. Users can formulate flexible queries that can be translated into such graphs. Alternatively, users can input query graphs by editing an abstract model (model template) derived either from a model database or from an example scene in the media database. Search and retrieval is accomplished by matching the query graph with those instantiated models in the database. The proposed approach allows for integrated textual and low-level, descriptor-based video search, where the integration is enabled by the model. Examples and experimental results are provided to demonstrate the effectiveness of the proposed integrated modeling and querying framework.