LIDH: An Efficient Filtering Method for Approximate k Nearest Neighbor Queries Based on Local Intrinsic Dimension

Abstract

Due to the so-called “curse of dimensionality” causing poor performance when querying in the high-dimensional space, the high-dimensional approximate kNN (AkNN) query has been extensively explored to trade accuracy for efficiency. In this paper, we propose a Local Intrinsic Dimension-based Hashing (LIDH) method for the high-dimensional AkNN query which locates a definite searching range by Local Intrinsic Dimensionality for filtering data points. Specifically, we propose a filter-refinement model for the AkNN query to avoid the virtual rehashing with fewer index space. Experimental evaluations demonstrate that our method can provide higher I/O and CPU efficiency while retaining satisfactory query accuracies.