High Concurrent Elastic Resource Allocation in Hadoop YARN

Abstract

Efficient resource management to improve the throughput in large-scale cluster has become a research focus with the rapid development of applications of Big Data. YARN (Yet Another Resource Negotiator), as the new generation of resource management system in Hadoop, is more efficient in resource utilization and capable of handling more kinds of workload than previous systems. Due to the fact that a task usually occupies more resources than it actually uses during some stage of its life cycle, a relevant amount of resource is idle and can not be allocated to satisfy the requirements of pending tasks. In order to address the deficiencies of resource allocation in YARN, this paper presents a high concurrent elastic resource allocation strategy named Ballon, which can dynamically adjust the configured resource of a node depending on the actual resource utilization of the node. Moreover, Ballon classifies resource requests of applications into different types. Consequently the elastic resources can be allocated to proper request. Our experiments demonstrate that Ballon cluster can reduce the average execution time of application by at least 10% in most MapReduce application and can increase the resource utilization of cluster.