Multi-layer Resources Fair Allocation in Big Data with Heterogeneous Demands

Abstract

The resource fair allocation is very important for the most systems. This paper focuses on the resource allocation in a big data system. This system has the characteristics of multiple resources owning and heterogeneous resource demanding. Firstly, a basic allocation scheme is proposed. Some allocation properties are introduced including sharing incentive, strategy proofness, envy-freeness, Pareto optimality, resource efficiency, single-layer multi-resource fairness, multi-layer single-resource fairness. Secondly, a single layer dominant and max–min fair allocation (SDMMF) is proposed. The basic idea is that the resources are firstly allocated using dominant resource fairness scheme. If some resources are not fully allocated, using max–min fairness scheme allocate the surplus ones. The SDMMF is theoretically proved satisfying all the hierarchical resource allocation properties. Thirdly, a Multi-layer resource allocation scheme, namely multi-layer dominant and max–min fair allocation (MDMMF) is proposed. The basic idea is that a multi-layer resource demand hierarchy can be divided into a series of single layer resource demand. So a resource demand vector can be built in the hierarchy from down to top. Then, the resource allocation of first layer is calculated using SDMMF allocation. The result of such allocation is further allocating in the third layer. Recursively do the MDMMF allocation until the bottom layer nodes obtain resource allocation shares. Lastly, The MDMMF is also theoretically proved satisfying all the hierarchical resource allocation properties.