MISTRAL - measurable, intelligent and reliable semantic extraction and retrieval of multimedia data

Abstract

Multimedia data has a rich and complex structure in terms of inter- and intra-document references. Its potential is severely limited unless effective methods for semantic extraction and semantic-based cross-media exploration and retrieval can be devised. Today's leading-edge techniques in this area are working well for low-level feature extraction and, to a certain degree, for concept recognition and detection. However, they are focussing on narrow aspects of isolated collections of multimedia data, and are dealing only with single media types. In this paper we propose service oriented system architecture for measurable, intelligent and reliable semantic extraction and retrieval of multimedia data, short MISTRAL. MISTRAL improves on these aspects by extracting a large variety of semantically relevant metadata from one media type and integrating it with semantic concepts derived from other media types. Results from this cross-modal semantic integration shall be fed back into the semantic extraction processes for each different media type to further enhance the quality of their results. MISTRAL will focus on innovative, semantic-based cross-modal exploration and retrieval techniques employing concepts at different semantic levels. The specifics of multimedia data will be addressed in the global, networked context employing semantic web technologies including security issues in such a networked environment. On top of this, we propose to integrate ontologies into the extraction process. By using their inference capabilities a feedback mechanism to low level concept extraction modules will be provided, enabling MISTRAL to resolve ambiguities and further enrich extracted semantic concepts in a domain independent way. Retrieval results based on semantic and cross-modal techniques shall contribute to a significant improvement of human-computer interaction in multimedia retrieval and exploration applications.