A low-latency computing framework for time-evolving graphs

Abstract

The demand to deliver fast responses in processing time-evolving graphs is higher than ever before in a large number of big data applications. This problem promotes extensive uses of an incremental computing model, which executes the underlying graph algorithm on the newly updated graph structure by taking the results of the computation on the outdated graph structure as initial values, in distributed time-evolving graph computing systems. In this paper, we experimentally study how the initial values of the computation on a newly updated graph structure influence the convergence of the iterative graph analysis, and we develop an optimization framework on the basis of the incremental computing model to accelerate the convergence of processing time-evolving graphs thus achieving high performance for time-evolving graph analysis. In contrast to the traditional incremental computing model, which uses the results of the computation on the outdated graph structure directly, the proposed framework predicts the optimal initial values of the computation on the new graph structure and thereby reduces the number of iterations. Two different prediction approaches are designed to optimize the initial values based on a combination of the results of the computation on the previous graph data and the newly incoming graph data. We have evaluated our optimization framework using the graph algorithms PageRank and KMeans on Amazon EC2 clusters. The experiments demonstrate that the incremental computing implementation with the initial value prediction have reduced the number of iterations by 30% for the PageRank algorithm and 13.7% for the KMeans algorithm, and reduced the response time by 12.7% and 10.6% accordingly compared to the traditional incremental computing model.