Fair and near-optimal coflow scheduling without prior knowledge of coflow size

Abstract

Achieving the minimum average coflow completion time(CCT) and the isolation guarantees for multi-tenant, is considered a challenge in a cloud environment. This is because the minimum average CCT and isolation guarantees are two conflicting targets, and they cannot be achieved simultaneously. Prior solutions have implemented a single target either minimizing the average CCT or isolation guarantees. The prior solutions are also limited to clairvoyant scheduling. They also assume the availability of the complete knowledge of coflow sizes before the communication starts. In this paper, we propose an efficient scheduling algorithm smallest-height-first DRF(SHFDRF) for near-optimal scheduling and isolation guarantees without prior knowledge of coflow size. SHFDRF achieves the long-term isolation guarantees and the minimum average CCT by the smallest height first and the monopolistic dominant resource fairness bandwidth allocation strategy. The smallest height first and the monopolistic dominant resource fairness bandwidth allocation strategy can also improve link utilization and system throughput. The trace-driven simulation shows that SHFDRF enables communication stages to 1.28 $$\times $$ , 2.27 $$\times $$ , and 6.28 $$\times $$ faster on the 95th percentile compared to DRF, NCDRF, and Per-Flow Fairness. Even compared with minimum CCT, the completion time of coflow only slowed down by 13.9% on the 95th percentile. Overall, the performance of SHFDRF is acceptable, and it can be applied to the actual datacenter without the limitation of complete prior knowledge.