An intelligent annotation-based image retrieval system based on RDF descriptions

Abstract

In this paper, we aim at improving text-based image search using Semantic Web technologies. We introduce our notions of concept and instance in order to better express the semantics of images, and present an intelligent annotation-based image retrieval system. We test our approach on the Flickr8k dataset. From the provided captions, we generate annotations at three levels (sentence, concept and instance). These annotations are stored as RDF triples and can be queried to find images. The experimental results show that using concepts and instances to annotate images flexibly can improve the intelligence of the image retrieval system: (1) with annotations at concept level, it enables to create semantic links between concepts and then addresses many challenges, such as the problems of synonyms and homonyms; (2) with annotations at instance level, it can count things (e.g., “two people”, “three animals”) or identify a same concept.