Bandwidth-Aware Scheduling With SDN in Hadoop: A New Trend for Big Data

Abstract

Software-defined networking (SDN) is a revolutionary network architecture that separates out network control functions from the underlying equipment and is an increasing trend to help enterprises build more manageable data centers where big data processing emerges as an important part of applications. To concurrently process large-scale data, MapReduce with an open-source implementation named Hadoop is proposed. In practical Hadoop systems, one kind of issue that vitally impacts the overall performance is known as the NP-complete minimum make span problem. One main solution is to assign tasks on data local nodes to avoid link occupation since network bandwidth is a scarce resource. Many methodologies for enhancing data locality are proposed such as the Hadoop default scheduler (HDS) and state-of-the-art scheduler balance-reduce scheduler (BAR). However, all of them either ignore allocating tasks in a global view or disregard available bandwidth as the basis for scheduling. In this paper, we propose a heuristic bandwidth-aware task scheduler bandwidth-aware scheduling with SDN in Hadoop (BASS) to combine Hadoop with SDN. It is not only able to guarantee data locality in a global view but also can efficiently assign tasks in an optimized way. Both examples and experiments demonstrate that BASS has the best performance in terms of job completion time. To our knowledge, BASS is the first to exploit talent of SDN for job scheduling of big data processing and we believe that it points out a new trend for large-scale data processing.