Adding network bandwidth resource management to Hadoop YARN

Abstract

YARN (Yet Another Resource Negotiator) is an emerging resource management system in Hadoop platform which can process big data in the scale of petabytes/day. YARN, as a resource management system, needs manage all kinds of resources in the clusters, such as cpu, memory, disk io, network io and so on. At present, YARN mainly focuses on computing resource and storage resource (CPU and memory), while ignoring network management. But research shows that efficient network bandwidth resource management is crucial to optimize performance. In this paper we present a network bandwidth resource management mechanism based on YARN that takes into account the allocation and isolation of applications' network bandwidth. Experiments show that the proposed solution makes YARN run applications in a more predictable manner.