Multi-job Meta-brokering in Distributed Computing Infrastructures Using Pliant Logic

Abstract

The ever growing number of computation-intensive applications calls for the interoperation of distributed infrastructures such as Clouds, Grids and private clusters. The European SHIWA and ER-flow projects have been initiated to enable the combination of heterogeneous scientific workflows, and to execute them in a large-scale system consisting of multiple Distributed Computing Infrastructures including Grids and Clouds. In this paper we focus on one of the resource management challenges of these projects called multi-job scheduling. A parameter study job of a workflow having a large number of input files to be consumed by independent job instances is called a multi-job. In order to cope with the high uncertainty and unpredictable load of these infrastructures and with the simultaneous submissions of multi-job instances, we use statistical historical job allocation data gathered from real-world workflow archives and propose an adaptive meta-brokering approach for the management of this unified system based on the Pliant logic concept, which is a specific part of fuzzy logic theory. We argue that this novel scheduling technique produce better performance scores, hence the overall load of the system can be more balanced.