Exploiting lower bounds to accelerate approximate nearest neighbor search on high-dimensional data

Abstract

Approximate nearest neighbor (ANN) search in high dimensional space is a fundamental operation in many applications. During the past decades, many indexing methods have been proposed to solve ANN search, including inverted index based methods, tree structures, locality sensitive hashing based methods and proximity graphs.In this paper, we explore lower bound techniques to speed up ANN search over the existing indexing methods. Since traditional lower bounds cannot reach a satisfying tradeoff between efficiency and tightness, we examine the concept of the existing lower bounds and define two new types of lower bounds, i.e., progressive lower bound (PLB) and statistical lower bound (SLB), which expand the concept of lower bound. Moreover, we propose a good instance of PLB, i.e. progressive partial distance (PPD). As the first attempt, we propose the idea of SLB and demonstrate asymmetric quantizer distance (AQD) as a good instance of SLB. Notably, AQD was only treated as an estimation of the distance in previous works, while we discover its functionality of being a lower bound in this paper.According to the extensive experiments on real data sets, we demonstrate that our lower bounds are able to obviously accelerate ANN search of the existing indexing methods, and our lower bounds outperform the existing lower bounds by a significant margin, due to their strong pruning powers. Especially, AQD speedups the state-of-the-art indexing method HNSW (Malkov and Yashunin, 2016) 1.9 times for a high recall 0.95. As to other indexing methods, the speedups of AQD are even higher, because they need to access more candidates than HNSW.