Sharing of Cluster Resources among Multiple Workflow Applications

Abstract

Many computational solutions can be expressed as workflows. A Cluster of processors is a shared resource among several users and hence the need for a scheduler which deals with multi-user jobs presented as workflows. The scheduler must find the number of processors to be allotted for each workflow and schedule tasks on allotted processors. In this work, a new method to find optimal and maximum number of processors that can be allotted for a workflow is proposed. Regression analysis is used to find the best possible way to share available processors, among suitable number of submitted workflows. An instance of a scheduler is created for each workflow, which schedules tasks on the allotted processors. Towards this end, a new framework to receive online submission of workflows, to allot processors to each workflow and schedule tasks, is proposed and experimented using a discrete-event based simulator. This space-sharing of processors among multiple workflows shows better performance than the other methods found in literature. Because of space-sharing, an instance of a scheduler must be used for each workflow within the allotted processors. Since the number of processors for each workflow is known only during runtime, a static schedule can not be used. Hence a hybrid scheduler which tries to combine the advantages of static and dynamic scheduler is proposed. Thus the proposed framework is a promising solution to multiple workflows scheduling on cluster.