Towards shorter task completion time in datacenter networks

Abstract

Datacenters are now used as the underlying infrastructure of many modern commercial operations, powering both large Internet services and a growing number of data-intensive scientific applications. The tasks in these applications always consist of rich and complex flows which require different resources at different time slots. The existing data center scheduling frameworks are however base on either task or flow level metrics. This simplifies the design and deployment, but hardly unleashes the potentials of obtaining low task completion time for delay sensitive applications. In this paper, we show that the performance (e.g., tail and average task completion time) of existing flow-aware and task-aware network scheduling is far from being optimal. To address such a problem, we carefully examine the possibility to consider both task and flow level metrics together and present the design of TAFA (Task-Aware and Flow-Aware) in data center networks. This approach seamlessly combines the existing flow and task metrics together while successfully avoids their problems as flow-isolation and flow indiscrimination. The evaluation result shows that TAFA can obtain a near-optimal performance and reduce over 35% task completion time for the existing data center systems.