Visual Arts Search on Mobile Devices

Abstract

Visual arts, especially paintings, appear everywhere in our daily lives. They are not only liked by art lovers but also by ordinary people, both of whom are curious about the stories behind these artworks and also interested in exploring related artworks. Among various methods, the mobile visual search has its merits in providing an alternative solution to text and voice searches, which are not always applicable. Mobile visual search for visual arts is far more challenging than the general image visual search. Conventionally, visual search, such as searching products and plant, focuses on locating images containing similar objects. Hence, approaches are designed to locate objects and extract scale-invariant features from distorted photos that are captured by the mobile camera. However, the objects are only part of the visual art piece; the background and the painting style are both important factors that are not considered in the conventional approaches. In this article, an empirical investigation is conducted to study issues in photos taken by mobile cameras, such as orientation variance and motion blur, and how they influence the results of the mobile visual arts search. Based on the empirical investigation results, a photo-rectification pipeline is designed to rectify the photos into perfect images for feature extraction. A new method is proposed to learn high discriminative features for visual arts, which considers both the content information and style information in visual arts. Apart from conducting solid experiments, a real-world system is built to prove the effectiveness of the proposed methods. To the best of our knowledge, this is the first article to solve problems for visual arts search on mobile devices.