PFT: A Performance-Fairness scheduler on Hadoop YARN

Abstract

Performance and fairness are two important factors for Hadoop cluster. Many previous studies either focus on the improvement of performance or fairness solely, and most of which are based on the first generation of hadoop(MRvl). However, few studies consider the tradeoff between the performance and the fairness on Hadoop YARN, the second generation of hadoop(MRv2). In this paper, we propose a new scheduling algorithm for Hadoop YARN, named PFT, which can effectively tradeoff the performace and the fairness, and reduce the makespan of MapReduce jobs by utilizing multi-level queue, time factor, job urgency factor, and domain resource ratio. We implement PFT as a pluggable scheduler in Hadoop YARN. Experimental results show that PFT can reduce the makespan of MapReduce jobs by 34.53% improve the CPU utilization by 35.93% and improve the memory utilization by 38.98%