Building and using fuzzy multimedia ontologies for semantic image annotation

Abstract

This paper proposes a methodology for building fuzzy multimedia ontologies dedicated to image annotation. The built ontology incorporates visual, conceptual, contextual and spatial knowledge about image concepts in order to model image semantics in an effective way. Indeed, our approach uses visual and conceptual information to build a semantic hierarchy that will serve as a backbone of our ontology. Contextual and spatial information about image concepts are then computed and incorporated in the ontology in order to model richer semantic relationships between these concepts. Fuzzy description logics are used as a formalism to represent our ontology and the inherent uncertainty and imprecision of this kind of information. Subsequently, we propose a new approach for image annotation based on hierarchical image classification and a multi-stage reasoning framework for reasoning about the consistency of the produced annotation. In this approach, fuzzy ontological reasoning is used in order to achieve a semantically relevant decision on the belonging of a given image to the set of concepts from the annotation vocabulary. An empirical evaluation of our approach on Pascal VOC'2009 and Pascal VOC'2010 datasets has shown a significant improvement on the average precision results.