Supervised reranking for web image search

Abstract

Visual search reranking that aims to improve the text-based image search with the help from visual content analysis has rapidly grown into a hot research topic. The interestingness of the topic stems mainly from the fact that the search reranking is an unsupervised process and therefore has the potential to scale better than its main alternative, namely the search based on offline-learned semantic concepts. However, the unsupervised nature of the reranking paradigm also makes it suffer from problems, the main of which can be identified as the difficulty to optimally determine the role of visual modality over different application scenarios. Inspired by the success of the learning-to-rank idea proposed in the field of information retrieval, we propose in this paper the paradigm, which derives the reranking function in a supervised fashion from the human-labeled training data. Although supervised learning is introduced, our approach does not suffer from scalability issues since a unified reranking model is learned that can be applied to all queries. In other words, a query-independent reranking model will be learned for all queries using query-dependent reranking features. The query-dependent reranking feature extraction is challenging since the textual query and the visual documents have different representation. In this paper, 11 lightweight reranking features are proposed by representing the textual query using visual context and pseudo relevant images from the initial search result. The experiments performed on two representative Web image datasets demonstrate that the proposed learning-to-rerank algorithm outperforms the state-of-the-art unsupervised reranking methods, which makes the learning-to-rerank paradigm a promising alternative for robust and reliable Web-scale image search.