Aggregating Uncertain Incast Transfers in BCube-Like Data Centers

Abstract

Many data-intensive applications like MapReduce are network-bound in data centers, due to transfer massive amount of flows across successive processing stages. Data flows in such an incast or shuffle transfer are highly correlated and aggregated at the receiver side. Prior work aims to aggregate correlated flows of each transfer, during the transmission phase as early as possible, so as to directly lower down the network traffic. However, many applications do not constrain the flows’ endpoints of each transfer as long as certain constraints are satisfied. Such uncertain transfers bring new opportunities and challenges to lower down the network traffic than prior deterministic transfers. In this paper, we focus on aggregating an uncertain incast transfer and minimizing the amount of caused network traffic. Prior approaches, relying on deterministic incast transfers, remain inapplicable. This paper makes the first step towards the study of aggregating uncertain incast transfer. We propose efficient approaches from two aspects, i.e., the initialization of uncertain senders and the incast tree building. We first design two initialization methods to pick the best deterministic senders for an uncertain incast transfer, so as to form the least number of disjoint sender groups. Thus, flows from each group would be aggregated as one flow on a common one-hop neighbor, irrespective of the location of a picked receiver. Accordingly, we propose the interstage-based and intrastage-based incast tree building methods to exploit the benefits of our initialization methods. We provide evidence to show that our approach can achieve the benefits of in-network aggregation for any uncertain incast transfer. Moreover, an uncertain incast transfer significantly outperforms any related deterministic one, in terms of the reduced network traffic and the saved network resources.