WISE: Workload-Aware Partitioning for RDF Systems

Abstract

Masses of large-scale knowledge graphs on various domains have sprung up in recent years. They are no longer able to be managed on a single machine. The distributed RDF systems intervene in the scalability issue using partitioning techniques. However, most of these systems are unaware of query workload and employ static partitioning. As diverse and dynamic workloads keep emerging in the knowledge graph applications, they cannot consistently provide good performance. To address the problem, we propose a workload-aware partitioning framework WISE, which could be deployed on any initial partitioning. It encodes the incoming SPARQL queries in a novel structure called query span and periodically examines the query span to identify the frequent query patterns. The triples of a frequent query pattern are moved to the same partition, aiming at improving the response time in the future. Our experiments on various RDF datasets and workloads indicate that WISE achieves dramatic communication reduction and considerable performance improvement over the baseline method. The migration overhead only accounts for a small portion of the total runtime.