Deadline-aware Preemptive Job Scheduling in Hadoop YARN Clusters

Abstract

As a popular open-source framework for big data processing, Hadoop Yarn has been widely used by large internet and e-commerce companies such as Amazon, Alibaba, and Facebook. One of the main challenges faced by Hadoop service providers is to optimize the scheduling of MapReduce jobs in order to provide desired performance levels to users and improve cluster resource utilization. In this paper, we propose a deadline-aware preemptive job scheduling strategy, named DAPS, to minimize the job deadline misses. In the proposed DAPS, the job scheduling problem is formulated as an online optimization problem, and a preemptive resource allocation algorithm is developed to search for a good job scheduling policy. We implement the proposed DAPS in Hadoop Yarn clusters and evaluate the effectiveness of the proposed DAPS through several experiments. Our proposed approach has an average job completion rate of 92.33%, which is better than capacity scheduler (62.94%), Earliest Deadline First scheduler (78.76%), and PDSonQueue (87.58%). The experimental results show that the performance of the proposed DAPS is superior to the state-of-the-art strategies applied to similar problems.