VSIM: Distributed local structural vertex similarity calculation on big graphs

Abstract

In many graph analytical applications, the local structural vertex similarity calculation is an essential prerequisite for advanced graph mining. The similarity calculation finds out all the similar vertex pairs whose local structural similarity scores (like the number of common neighbors, and the Jaccard index of adjacency sets) are above a given threshold. The real-world applications use a wide range of similarity thresholds. However, the existing distributed methods for the problem only optimize for either high thresholds (> 0.7) or low thresholds (< 0.1). To overcome the drawback, we propose a new distributed vertex similarity calculation framework VSIM that is efficient under a broad range of thresholds. VSIM processes static undirected graphs with local structural similarity scores that measure the similarity between vertices based on the first-order topology information. VSIM generates a similarity calculation task for every vertex in the graph and conducts all the tasks in parallel on a distributed computing platform along with a distributed key-value store. Each task finds vertices similar to a given center vertex with two task execution modes. The two modes optimize for high and low thresholds, respectively. Each task picks the suitable mode adaptively according to cost estimation models. We also propose an efficient implementation for VSIM on Apache Spark, with three optimization techniques to reduce communication costs and balance workloads on power-law graphs. The experimental evaluation shows that VSIM outperforms the state-of-the-art distributed methods by up to 67x speedup. VSIM can achieve near-linear node scalability in low-threshold and small cache scenarios.