PCSP: A Preemptive Capacity Scheduler Policy for Scheduling Hadoop Jobs

Abstract

Map reduce is a parallel programming paradigm used for processing massive data sets. A popular open-source implementation of Map reduce is Hadoop. There are basic schedulers embedded in Hadoop, including First in First out (FIFO), Fair Scheduler, and Capacity Scheduler (CS). Currently, researches have been focused on Capacity Scheduler to improve the Capacity Scheduler. Native Capacity Scheduler does not support the preemption, which results in the starvation caused by the non-preemptive scheduling. To resolve this problem, a Preemptive Capacity Scheduler Policy (PCSP) is proposed. Finally, we implement the PCSP on Hadoop, the experimental results of which indicate that PCSP we proposed is efficient in running Hadoop jobs.