Bridging the gap: effect of text query reformulation in multimodal retrieval

Abstract

Multimodal Retrieval provides new paradigms and methods aimed at effectively searching through the enormous volume of data. Multimodal retrieval is a well studied problem often used in image retrieval. Most of the existing works in image retrieval under the pretext of multimodality stress on bridging the semantic gap by using both textual and visual features. In this paper, we use relevance feedback from the user-generated documents associated with the images for expanding textual query and study its effect on both image and text retrieval. We employ a topic decomposition based keyphrase extraction technique to expand the textual queries. Our results articulate the fact that an insightful textual query expansion always improves retrieval performance for both textual or image retrieval. Also, we adopt optimum weight learning scheme to combine the modalities in a privileged way. We perform a comparative study with two well established keyphrase extraction techniques which are used for textual query expansion. A detailed set of experiments on a standard real world dataset is also carried out for the same.