On Max-min Fair Resource Allocation for Distributed Job Execution

Abstract

In modern data intensive computing, it is increasingly common for jobs to be executed in a distributed fashion across multiple machine clusters or datacenters to take advantage of data locality. This paper studies fair resource allocation among jobs requiring distributed execution. We extend conventional max-min fairness for resource allocation in a single machine or machine cluster to distributed job execution over multiple sites and define Aggregate Max-min Fairness (AMF) which requires the aggregate resource allocation across all sites to be max-min fair. We show that AMF satisfies the properties of Pareto efficiency, envy-freeness and strategy-proofness, but it does not necessarily satisfy the sharing incentive property. We propose an enhanced version of AMF to guarantee the sharing incentive property. We present algorithms to compute AMF allocations and propose an add-on to optimize the job completion times under AMF. Experimental results show that compared with a baseline which simply requires the resource allocation at each site to be max-min fair, AMF performs significantly better in balancing resource allocation and in job completion time, particularly when the workload distribution of jobs among sites is highly skewed.