Knowledge representation and semantic annotation of multimedia content

Abstract

Knowledge representation and semantic annotation of multimedia documents typically have been pursued in two different directions. Previous approaches have focused either on low-level descriptors, such as dominant colour, or on the semantic content dimension and corresponding manual annotations, such as person or vehicle. Here, a knowledge infrastructure and an experimentation platform for semantic annotation to bridge the two directions are presented. Ontologies are being extended and enriched to include low-level audiovisual features and descriptors. Additionally, a tool that allows for linking low-level MPEG-7 visual descriptions to ontologies and annotations is presented. Thus, ontologies that include prototypical instances of high-level domain concepts together with a formal specification of the corresponding visual descriptors are constructed. This infrastructure is exploited by a knowledge-assisted analysis framework that may handle problems such as segmentation, tracking, feature extraction and matching in order to classify scenes, identify and label objects and thus automatically create the associated semantic metadata.