On the coupled use of signal and semantic concepts to bridge the semantic and user intention gaps for visual content retrieval

Abstract

The effectiveness of image indexing and retrieval systems has been negatively impacted by both the semantic and user intention gaps. The first is related to the challenge of characterizing the visual semantic information through low-level extracted features (color, texture...) while the second highlights the difficulty for human users to convey their search intents using the traditional relevance feedback or query-by-example visual query mechanisms. We address both issues by highlighting vocabularies of visual concepts that are mapped to extracted low-level features through an automated learning paradigm. These are then instantiated within a semantic indexing and retrieval framework based on a Bayesian model considering the joint distribution of visual and semantic concepts. To address the user intention gap and enrich the expressiveness of the retrieval module, visual and semantic concepts can be coupled within text-based queries. We are therefore able to process both single-concept queries such as the state-of-the-art solutions but also topic-based queries, i.e. non-trivial queries involving multiple characterizations of the visual content. We evaluate our proposal in a precision/recall based evaluation framework on the IAPR-TC 12 benchmark dataset.