A large-scale graph partition algorithm with redundant multi-order neighbor vertex storage

Abstract

Recently, graph data become increasingly important and larger in many fields, and many distributed graph computing systems have been proposed to deal with large-scale graphs. The graph partition algorithm is the basis of these systems. In many graph applications, a vertex updates its feature by aggregating information of its multi-order neighboring vertices, and the existing graph partition algorithms often lead to heavy communication cost. This paper proposes a graph partition algorithm to store the multi-order neighbor vertices redundantly to avoid the communication requirement. It first formulates the problem of graph partitioning with redundant multi-order neighbor vertices storage as an optimization problem, and then proposes PARN (partition algorithm with redundant multi-order neighbor) algorithm to tackle it. The PARN algorithm consists of initial partition and vertex migration based on genetic algorithm, and it classifies the vertices of a partition into primary and auxiliary vertices. The auxiliary vertices required for the primary vertices in a partition are stored in the same partition, so the primary vertices don't need to fetch the information from the other partitions. The experiment results show that the proposed algorithm is comparable to Hash, GAP, GNP, and QCLP algorithms in terms of load balance and produces fewer auxiliary vertices than these algorithms.