Distributed online similarity search in high dimensional space

Abstract

In this paper, we consider distributed on-line similarity search for big data in high dimensional spaces, for which Locality Sensitive Hashing (LSH) was the method of choice. But LSH scheme needs a rather large number of hash tables and optimal parameters. So, it is difficult for LSH to deal with big data in one machine. To reduce the size of big data, we divide the dataset into well separated clusters with bounded aspect ratios, locating them in different peers in ring network, using random projection tree(RP-tree). To limit the number of network accesses, we put similar subgroups adjacent to each other. Then, we construct one LSH hash table for each subgroup using optimal parameters. It is shown by comprehensive performance evaluations using real world data that our approach decreases the network cost and brings major performance improvement, while maintaining a good load balance between different machines.