Scheduling MapReduce Jobs Under Multi-round Precedences

Abstract

We consider non-preemptive scheduling of MapReduce jobs consisitng of multiple map-reduce rounds so as to minimize the average weighted completion time on identical and unrelated processors. For identical processors, we present LP-based O1-approximation algorithms, while for unrelated processors the approximation ratio naturally depends on the maximum number of rounds of any job a small constant in practice. For the single-round case, we substantially improve on previously best known approximation ratios for both identical and unrelated processors. Moreover, we conduct an experimental analysis and compare the performance of our algorithms against a fast heuristic and a lower bound on the optimal solution, thus demonstrating their promising practical performance.