Dynamic Resource Provisioning for Iterative Workloads on Apache Spark

Abstract

Apache Spark as a popular in-memory data analytic framework has been employed by various applications---such as machine learning, graph computation, and scientific computing, which benefit from the long-running process (e.g. executor) programming model to avoid system I/O overhead. Since the resource usages of long-running applications like iterative computation vary significantly over time, we find that peak demand based resource allocation policies lead to low cloud utilization in production environments. In this paper, we present a utilization aware resource provisioning approach for iterative workloads on Apache Spark (iSpark). iSpark aims to timely scale up or scale down the number of executors in order to fully utilize the allocated resources while taking the dominant factor into consideration. Testbed evaluations show that iSpark averagely improves the resource utilization of individual executors by 35.2% compared to vanilla Spark. Furthermore, we have extended iSpark to multi-tenancy cloud environments. Specifically, we extend the two-dimensional resource constraints (i.e., CPU and MeM) in iSpark to three-dimensional resource constraints (i.e., CPU, MeM and I/O) to include I/O performance in the cloud environment. Experimental results on virtual clusters with varying interferences show that iSpark with cloud extension improves the average job completion time by 68% compared to the default policy.