PROSE: Multi-round fair coflow scheduling without prior knowledge

Abstract

Coflow scheduling plays a crucial role in enhancing network-level performance in datacenter applications. Existing works on coflow scheduling predominantly concentrate on clairvoyant schedulers, which assume prior knowledge of coflow sizes before transmission. However, this assumption is often unrealistic in practical scenarios like pipelined computation. Consequently, research on non-clairvoyant schedulers has garnered increasing attention. Most studies on non-clairvoyant schedulers focus on either improving efficiency or promoting fairness, with limited emphasis on striking a balance between the two objectives. Additionally, for non-cooperative environments like multi-tenant datacenter networks, it is essential to ensure strategy-proofness in coflow scheduling. Without adequate measures, tenants may manipulate their reported requirements to gain an unfair advantage in resource allocation. Ideally, in non-cooperative environments, a coflow scheduler should achieve efficiency, fairness, and strategy-proofness, all without prior knowledge of coflow sizes. However, few existing efforts have managed to simultaneously achieve these objectives.To fill this gap, we develop PROSE, a non-clairvoyant scheduler for non-cooperative environments. PROSE divides the scheduling time into multiple rounds and employs a two-stage scheduling mechanism based on these rounds. In the first stage, PROSE estimates the sizes of the coflows with unknown sizes by transmitting the probe flows, using a policy that combines coflow competition with batch processing. Once the information about coflow sizes is obtained, PROSE prioritizes coflows among tenants for bandwidth allocation in the second stage, aiming to accelerate coflow completion within each round. Furthermore, PROSE ensures optimal isolation guarantee by fairly limiting the total data transfer for each tenant and dynamically adjusts the scheduling order of coflows in a round-robin manner across multiple rounds. PROSE prevents tenants from attempting to consume excessive network resources through manipulative actions that deceive the scheduler. Trace-driven simulations demonstrate that PROSE outperforms the fair scheduler, achieving up to a 2.37× reduction in average coflow completion time, indicating substantial efficiency improvements.