An Improved Adaptive Space-Sharing Scheduling Policy for Non-dedicated Heterogeneous Cluster Systems

Abstract

Adaptive space-sharing scheduling algorithms tend to improve the performance of clusters by allocating processors to jobs based on the current system load. The focus of existing adaptive algorithms is on dedicated homogeneous and heterogeneous clusters. However commodity clusters are naturally non-dedicated and tend to be heterogeneous over the time as cluster hardware is usually upgraded and new fast machines are also added to improve cluster performance. The existing adaptive policies for dedicated cluster systems are not suitable for such conditions. Moreover existing adaptive policies use First-come-first-serve (FCFS) which is known to be sensitive of variance in service demand, as a job-selection policy for processor allocation. FCFS allocation of processors to jobs results in a situation where small jobs could be blocked by an earlier arrived large job. This paper fills these gaps by designing an efficient adaptive space-sharing scheduling algorithm for non-dedicated heterogeneous cluster systems. Evaluation results show that the proposed algorithm provide substantial improvement over existing algorithms at moderate to high system utilizations.