Deep learning hashing for mobile visual search

Wu Liu,Zhineng Chen,Dong Zhao,Heng Qi,Huadong Ma

doi:10.1186/s13640-017-0167-4

Abstract

The proliferation of mobile devices is producing a new wave of mobile visual search applications that enable users to sense their surroundings with smart phones. As the particular challenges of mobile visual search, achieving high recognition bitrate becomes the consistent target of existed related works. In this paper, we explore to holistically exploit the deep learning-based hashing methods for more robust and instant mobile visual search. Firstly, we present a comprehensive survey of the existed deep learning based hashing methods, which showcases their remarkable power of automatic learning highly robust and compact binary code representation for visual search. Furthermore, in order to implement the deep learning hashing on computation and memory constrained mobile device, we investigate the deep learning optimization works to accelerate the computation and reduce the model size. Finally, we demonstrate a case study of deep learning hashing based mobile visual search system. The evaluations show that the proposed system can significantly improve 70% accuracy in MAP than traditional methods, and only needs less than one second computation time on the ordinary mobile phone. Finally, with the comprehensive study, we discuss the open issues and future research directions of deep learning hashing for mobile visual search.

Highlights

The proliferation of increasingly capable mobile devices opens up exciting possibilities for massive mobile applications
Mobile visual search, which can utilize mobile device to sense and understand what the users are watching at any time from any place, plays a key role in these applications
We comprehensively investigate the possibility of exploiting the deep learning hashing for mobile visual search

Summary

Introduction

The proliferation of increasingly capable mobile devices opens up exciting possibilities for massive mobile applications. Mobile visual search, which can utilize mobile device to sense and understand what the users are watching at any time from any place, plays a key role in these applications. The always-on broadband connection makes users always online. The abundant sensors can accurately supply sufficient and effective information for mobile perception. The increased computational ability of mobile device can instantly process the sensed information and fetch the related feedback. We can conveniently sense where we are [1], what we are watching [2,3,4] or what happened with our surrounding [5, 6] with mobile visual search immediately

Methods

Findings

Discussion

Conclusion