Multiple resources scheduling for diverse workloads in heterogeneous datacenter

Abstract

Multiple resources allocation in large scale datacenter has been widely studied and used in recent years. However, the existing scheduling methods have some shortcomings in the heterogeneous environment, such as starving certain jobs, leaving some important resources unallocated, and decreasing system throughput. In this paper, we generalize the most popular multi-resource scheduling algorithm, Dominant Resource Fairness (DRF), to the heterogeneous datacenter, and propose the Heterogeneous DRF (H-DRF) algorithm. In H-DRF, we first introduce the parameter of weight for coprocessors to avoid job starvation and important resources unallocated. Second, we consider the factors of running tasks and average executing time of each job to ensure the fairness and throughput of the system. Finally, we implement H-DRF algorithm in the YARN resource manager for scheduling diverse workloads. The experimental results show that our proposed algorithm, H-DRF, leads to higher resources utilization and better throughput than DRF sharing scheme in heterogeneous datacenter.