Slick: A Coordinated Job Allocation Technique for Inter-Grid Architectures

Abstract

Large scale Grid computing systems are often organized as an inter-Grid architecture, where multiple Grid domains are interconnected through their local broker. In this context, the main challenge is to devise appropriate job scheduling policies that can satisfy goals such as global load balancing together with maintaining the local policies of the different Grids. This paper presents SLICK, a scalable resource discovery and job scheduling technique for broker based interconnected Grid domains. In this technique we leave local scheduling policies untouched, while inter-Grid scheduling decisions are handled by a separate scheduler installed on local brokers. To make suitable scheduling decisions, brokers must collect information about current resource usage at other domains. To this end, brokers periodically exchange their local domain's resource usage information with their neighbors. For large scale systems, this periodic exchange naturally leads to a significant amount of traffic. To avoid that the broker overlay becomes overloaded, we introduce an aggregation technique to reduce and combine worker resource usage information. We have compared SLICK with three other techniques through simulation of 50,000 node Grid divided into 512 domains. We used synthetic job sequences with a total load of 80,000 jobs. Our results show that SLICK is better at maintaining the overall throughput and load balancing than previous techniques.