Adaptive dual-criteria task group allocation for clustering-based multi-workflow scheduling on parallel computing platform

Abstract

Workflow scheduling has long been an important research topic in the field of parallel computing. Clustering-based methods are one of the major types of workflow scheduling approaches which have been shown superior to other kinds of methods in many cases due to their advantage of minimizing inter-task communication costs. Most previous research dealt with single workflow scheduling and focused on how to cluster the tasks within a workflow into a set of task groups. Recent research showed that utilizing idle time gaps between scheduled tasks is a promising direction for efficient multiple workflow scheduling. Since executing multiple workflows simultaneously is an inevitable need in modern shared parallel computing platforms, efficient task group allocation becomes a critical issue. In this paper, we studied such issue and proposed an innovative dual-criteria task group allocation method which considers both task group's finish time and potential resource utilization to effectively improve overall multi-workflow execution performance. In addition, an adaptive task group rearrangement mechanism was adopted to further improve performance. The proposed method has been evaluated with a series of simulation experiments and compared to previous approaches. The experimental results show that our method outperforms previous approaches across different workload conditions and workflow properties in terms of average makespan. The performance improvement ranges from 5 to 29 % for different conditions, achieving the largest performance improvement for workflows of smaller CCR.