A Visual Multimedia Query Language for Temporal Analysis of Video Data

Abstract

The storage of various media in multimedia databases poses new challenges to query techniques — challenges that exceed the expressive power of traditional text-based query languages. New query interfaces should take advantage of characteristics inherent in multimedia data, such as the dynamic temporal nature of video, the visual and spatial characteristics of images, the pitch of audio, etc. The focus of our research is to exploit the temporal continuity and combined spatio-temporal characteristics of video data for the purpose of video analysis. We do so by integrating a visual query paradigm with a dynamic visual presentation of results into a user-friendly interactive visualization environment. In this chapter, we present our overall approach for identifying trends in video data via querying for relationships between video annotations. Our approach allows users to analyze the video in terms of temporal relationships between events (e.g., events of type A frequently follow events of type B). We present a temporal visual query language (TVQL) for specifying relative temporal queries between sets of annotations. This query language builds on the notion of dynamic query filters and significantly extends them. It is tailored for temporal analysis — allowing users to pose queries, as well as to browse the data in a temporally continuous manner, thereby aiding them in the discovery of temporal trends. The TVQL is augmented with complementary temporal diagrams, which provide intuitive visual feedback for quickly and qualitatively verifying the temporal query specified. This chapter includes the complete specification of our TVQL, a transformation function for generating corresponding temporal diagrams, and the process for mapping TVQL queries to system queries.