A Two-Tier Approach to Grid Workflow Scheduling

Abstract

Contemporary workflow scheduling strategies that employ full-ahead planning of the complete workflow graph are typically designed for a resource model with high availability and good control over the resources by the scheduler. In pro- duction Grids with high utilization and autonomous sites, meta- scheduling is mostly focused on single jobs. Workflow scheduling strategies for Grids usually require continuous rescheduling during workflow execution or schedule single workflow tasks only. In this paper, we describe a methodology that combines existing scheduling strategies for the Grid and for workflows, and that operates under the constraints of production environments. It employs a list scheduling heuristic to create a full-ahead schedule of tasks, which are then distributed just-in-time according to resource performance predictions. I. INTRODUCTION In this paper, we propose an approach for the scheduling of Grid workflows within the MediGRID project (1) which is part of the German e-Science initiative D-Grid (2). MediGRID is a community Grid for researchers in the fields of medicine, biomedical informatics, and life sciences. Four types of pilot applications were selected for the first phase of MediGRID: bioinformatics, image processing, biomedical ontology, and clinical research applications. MediGRID has set up a ser- vice Grid that uses the Globus Toolkit 4 (GT4) (3) as its Grid middleware. On top of GT4, MediGRID employs an advanced workflow system for orchestrating the distributed execution of workflow applications on Grid resources. The important characteristic of these applications is that they are not monolithic and single-executable applications, but incor- porate multiple dependent computational modules, and entail transfer and storage of a large amount of data. The execution of these applications involves the concurrent and sequential execution of multiple programs, and the automatic and timely data transfer between programs which are also called tasks in workflow terms. A very important issue in executing a scientific workflow application in computational Grids is how to map and schedule workflow tasks onto multiple distributed resources, and how to handle task dependencies in a timely manner to deliver the requested performance. We describe a methodology to schedule application work- flows that combines existing scheduling strategies for work- flow tasks and Grid jobs in a performance-effective approach with manageable complexity. This approach consists of two tiers. In the first tier, a full-ahead schedule of the workflow tasks is created. This means that in a first step the whole workflow is scheduled statically before its execution. For this, we use a list scheduling heuristic similar to the Heterogeneous Earliest-Finish-Time algorithm (HEFT) (4). In the resulting static schedule, all tasks are assigned ranks which determine static prioritizations. The second tier operates dynamically at runtime and processes the workflow tasks according to their priorities. It performs a just-in-time mapping of tasks to Grid resources and is equivalent to common meta-scheduling on the Grid. To improve the just-in-time scheduling decision, it uses dynamic resource information and short-time predictions.