Core-set based product quantization for large-scale multimedia search

Abstract

Approximate nearest neighbor (ANN) search has been well-studied and weightily applied in the field of large scale multimedia search. In recent years, Product Quantization (PQ) based methods have achieved greate success in ANN search. However, both the time and space complexity for PQ learning are linear to the data amount, making it infeasible for large-scale datasets. In this paper, we propose a more efficient learning strategy for product quantization. Instead of learning PQ on the whole dataset, we construct a small set of representative elements, from which PQ can be approximated learned with much lower computation overhead. Extensive experiments are conducted on three large-scale datasets. The codebook learning process of our approach can achieve up to tens times speed-up than existing PQ-based methods, while the memory consumption is only logarithmic to the number of data points. At the same time, the ANN search accuracy of our method is still comparable with state-of-the-art methods on all datasets.