Low Complexity Hierarchical Scheduling for Diverse Datacenter Jobs

Abstract

Datacenter jobs in a private cloud are often generated by different departments or groups of a business or other organization. In order to reflect the organizational structures of datacenter workloads, it is desirable for the datacenter scheduler to support hierarchical scheduling. The state-of-the-art hierarchical scheduling scheme, hierarchical dominant resource fairness (H-DRF), however, is expensive to implement. Specifically, the worst-case time complexity of H-DRF is $O(q^{2})$ , where $q$ is dominated by the number of jobs in the hierarchy, which is a large complexity in a practical large-scale datacenter. In this letter, we design a new online hierarchical fair scheduling scheme, multi-resource collapsed hierarchies (MCH), in the way of converting the hierarchy into a weighted flat tree. MCH takes at most $O(q)$ time and is simple enough to implement in practice. Simulations driven by Google cluster traces show that MCH consumes a nearly sqrt of the total run time of H-DRF, improving significantly in saving the computation time.