On Efficiency of Multi-job Grid Allocation Based on Statistical Trace Data

Abstract

The ever growing number of computation-intensive applications calls for utilizing large-scale, potentially interoperable distributed infrastructures. Nowadays, such distributed systems enable the management of heterogeneous scientific workflows of considerable sizes, where job scheduling and resource management is a crucial issue. In this paper we focus on the challenges of scheduling parameter sweep applications, a specific and commonly used type of workflows where ordering of job executions is irrelevant. A parameter sweep has a large set of independent job instances, called a multi-job, submitted for execution in a single step. In order to cope with the high uncertainty and unpredictable load of resources, and the simultaneous submissions of multi-job instances, we propose a statistics-based brokering approach for allocating jobs to resources so that the makespan is minimised. Earlier studies claim that users’ predictions on job runtime are inaccurate and unusable for scheduling. Our aim is to examine, whether statistical trace data for the same purpose is efficient compared to randomized allocation.