Generating synthetic task graphs for simulating stream computing systems

Abstract

Listen

Translate article

Stream-computing is an emerging computational model for performing complex operations on and across multi-source, high-volume data flows. The pool of mature publicly available applications employing this model is fairly small, and therefore the availability of workloads for various types of applications is scarce. Thus, there is a need for synthetic generation of large-scale workloads to drive simulations and estimate the performance of stream-computing applications at scale. We identify the key properties shared by most task graphs of stream-computing applications and use them to extend known random graph generation concepts with stream computing specific features, providing researchers with realistic input stream graphs. Our graph generation techniques serve the purpose of covering a disparity of potential applications and user input. Our first “domain-specific” framework exhibits high user-controlled configurability while the second “application-agnostic” framework focuses solely on emulating the key properties of general stream-computing systems, at the loss of domain-specific fine-tuning.