Mobile image search with multimodal context-aware queries

Abstract

The proliferation of camera-equipped mobile devices with enhanced mobile computing power and network connectivity results in a rising demand for mobile image search. Although image search has been studied extensively over the last few decades, most existing solutions are based on and optimized for desktop and server platforms, not for mobile devices. In this paper, we address some of the challenging issues unique in mobile search scenarios and suggest a list of potential solutions. As a case study, we design a mobile landmark image search system to evaluate the effectiveness of some proposed solutions. To enhance the mobile search experience, we propose a multimodal search scheme which uses both image content and user location to increase search precision and thus minimize network usage. We also suggest a post-search result pruning method designed to match the most relevant results to a user's search interests. Experiments conducted on our Landmark-450 image dataset demonstrate that the proposed methods can significantly increase the relevance of selected results in response to mobile image search while also reducing the amount of data transferred across the network.