On MapReduce Scheduling in Hadoop Yarn on Heterogeneous Clusters

Abstract

Hadoop is a distributed computing system widely used for big data processing in various domains. As the data volume continues to increase rapidly, Hadoop systems have become a critical contributor to the success of many big data applications. The MapReduce scheduler is a key component that determines the overall performance of a Hadoop cluster. In this paper, we formulate and investigate a task scheduling problem in a heterogeneous Hadoop cluster to minimize the completion time of a batch of MapReduce jobs. We first design a prediction model to predict the end time of a task, which is used for placing the corresponding data block on a node in advance to reduce the data transmission time and the overall job completion time. Based on this prediction model, we propose a task matching-based scheduling algorithm, referred to as TMSA, to schedule the tasks in the task queue in Hadoop, by taking into account the real-time performance of each node in the cluster and the matching degree between nodes and tasks. Experimental results show that the prediction model achieves high accuracy and TMSA significantly reduces the completion time of a batch of MapReduce jobs compared to existing schedulers.