SARS: Towards minimizing average Coflow Completion Time in MapReduce systems

Abstract

Reducing the average Coflow Completion Time (CCT) plays a vital role to improve the responsiveness and throughput of distributed computing systems. However, most of the previous works on minimizing the average CCT scheduled all coflows following the Shortest Remaining Time First (SRTF) principle. In this paper, we argue that SRTF is not a necessary (even not a good) principle to follow in order to minimize the average CCT. Instead, we propose a Simulated Annealing based Reducer placement and coflow Scheduling approach named SARS to minimize the average CCT in MapReduce systems. In SARS, two powerful modules, i.e., the Neighboring State Searching (NSS) module and the System Energy Estimation (SEE) module, are designed to search a better coflow scheduling order and estimate the minimum average CCT given the scheduling order, respectively. Based on the interaction between these two modules, a near-optimal reducer placement and coflow scheduling scheme will be derived. Both testbed experiments and extensive simulations show that SARS can reduce the average CCT by up to 64.32% compared with the state-of-the-art technique.