A New Graph-Partitioning Algorithm for Large-Scale Knowledge Graph

Abstract

Large-scale knowledge graph is finding widely practical applications in many fields such as information retrieval, question answering, health care, and knowledge management and so on. To carry out computations on such large-scale knowledge graphs with millions of entities and facts, partitioning of the graphs is necessary. However, the existing partitioning algorithms are difficult to meet the requirements on both partition efficiency and partition quality at the same time. In this paper, we utilize the community-based characteristic that real-world graphs are mostly power-law distribution, and propose a new graph-partitioning algorithm (called MCS) based on message cluster and streaming partitioning. Compared with the traditional algorithms, MCS is closer to or even surpasses Metis package in the partition quality. In the partition efficiency, we use the PageRank algorithm in the spark cluster system to compute the Twitter graph data, and the total time of MCS is lower than that of Hash partitioning. With an increasing number of iterations, the effect is more obvious, which proves the effectiveness of MCS.