PERIDOT: Modeling Execution Time of Spark Applications

Abstract

A data analytics application submitted to a Spark cluster often has to finish executing by a specified deadline. To use cluster resources effectively, the key challenge is having the ability to gain quick insights on how the execution time of any given application is likely to be impacted by the resources allocated to the application, e.g., the number of Spark executor cores and the size of the input data. Such insights can be used to quickly estimate the required resources needed for the desired execution time. Our paper proposes an automated execution time estimation approach called PERIDOT that involves executing a given application under a fixed resource setting with two small subsets of its input data to offer fast, lightweight execution time predictions. It analyzes these two executions to estimate the internal dependencies of the application and combines them with knowledge of Sparks data partitioning mechanisms to derive an analytic model that can estimate execution times for other resource settings and input data sizes. Our results from a wide range of applications and multiple Spark clusters show that PERIDOT can accurately estimate the execution time of an application from limited historical data, and suggest the minimum amount of resources required to meet an execution deadline.