Scheduling strategy for computational-intensive data flow in generalized cluster environments

Abstract

A classical problem in the field of distributed computation and parallel processing concerns reasonable allocation of resources among computational-intensive data flows. We introduce the generalized cluster in this paper for processing large-scale scientific computations and to further explore a consecutive cooperation game-based dynamic scheduling strategy. We construct an abstract generalized cluster environment and summarize the types of data flows. We then convert the multi-objective scheduling problem into a multi-objective expectation function-based continuous cooperation game model and discuss its strategy and a solution for its kernel. We also propose a dynamic scheduling mechanism to address the instability of generalized clusters to ensure a reasonable, real-time adjusting allocation scheme by monitoring and compensating appropriately. Finally, we apply our method to a real world application to demonstrate our successful scheduling strategy that achieves superior results for overall cost, cost-performance index, and mean run time when compared with other methods.