Effective product quantization-based indexing for nearest neighbor search

Abstract

Product quantization is a widely used lossy compression technique that can generate high quantization levels by a compact codebook set. It has been conducted in cluster-based index structures, termed as product quantization-based indexing. In this paper, we propose a novel product quantization-based indexing method for approximate nearest neighbor search. Inspired by the study for learning to rank, a ranking scheme is presented to learn the weighting relation between query-dependent features. The clusters in an index table are ranked by the relevance scores derived from the weighted features with respect to the query. We then present an approximate nearest neighbor search algorithm integrating the proposed ranking scheme with the product quantization-based index structure. Experimental results on the billion-level datasets demonstrate the effectiveness and superiority of the proposed method compared with several state-of-the-art methods.