ARES: Aggressive Replication Enabled Scheduler for Hadoop Systems

Abstract

We re-examine the reliability issue in current Hadoop systems, and find that either rescheduling or speculative execution could possibly extend the job's deadline. Compared with the not long execution time of a job, the delay would be severe. To reduce such negative impact on execution time, we explore to adopt aggressive replication in scheduler, and present the design of ARES: Aggressive Replication Enabled Scheduler for Hadoop systems. Firstly, relying on the predictions on future coming workload, ARES could prevent denial of service caused by large consumption of resources. Secondly, with taking into account fairness and data locality, replicas of map tasks could be distributed fairly and would not increase much network latency. Finally, experiments show that using ARES could even shorten the job execution time because it not only improves the data locality, but also starts reduce tasks as soon as the fastest map replica is finished.