Webly-supervised zero-shot learning for artwork instance recognition

Abstract

This paper describes experiments on supervised approaches to webly-labeled artwork instance recognition and zero-shot learning for unseen artwork instance recognition. We build on our earlier work on webly-supervised learning using the NoisyArtdataset. The dataset consists of more than 90,000 images and in more than 3,000 webly-supervised classes, and a subset of 200 classes with verified test images. Document embeddings are provided for short descriptions of all artworks. NoisyArt is designed to support research on webly-supervised artwork instance recognition, zero-shot learning, and other approaches to visual recognition of cultural heritage objects. We report results of experiments on artwork instance recognition using the NoisyArt dataset of webly-labeled images as well as on the CMU-Oxford Sculptures dataset. In addition, we perform extensive experiments on zero-shot learning using webly-labeled training images for unseen artwork recognition. Our results demonstrate the benefits and limitations of zero-shot learning for instance recognition over webly-supervised data.