Leveraging multiple coflow attributes for information-agnostic coflow scheduling

Abstract

Recently, designing information-agnostic coflow scheduling mechanisms attracts much attention since by leveraging priority queues, they could reduce coflow completion time in data-parallel clusters without a priori knowledge, such as flow size, coflow size. However, existing information-agnostic mechanisms generally schedule coflows only according to the sent data size of different coflows and ignore other useful coflow-level attributes like width, length and communication patterns. In this paper, we investigate that the coflow completion time could be further decreased by jointly leveraging multiple coflow-level attributes. Based on this investigation, we present a Multiple-attributes-based Coflow Scheduling (MCS) mechanism to reduce the coflow completion time. In MCS, a Shortest and Narrowest Coflow First (SNCF) algorithm is designed to separate coflows based on their widths and estimated lengths at the start of a coflow. During the transmission of coflows, one type of demotion thresholds employed in previous coflow scheduling mechanisms is too crude for various coflows. Therefore, we proposed a double-threshold scheme to adjust the priorities of narrow (small coflow width) and wide (large coflow width) coflows according to different thresholds. Trace-driven simulations with production workloads show that MCS outperforms the previous information-agnostic scheduler Aalo, and reduces the coflow completion time of small coflows.