An enhanced priority-based scheduling heuristic for DAG applications with temporal unpredictability in task execution and data transmission

Abstract

Abstract In the context of large-scale distributed computing environments, among various DAG scheduling heuristics aiming at coping with severe uncertainties during task execution, those ones that try to maximize the parallelism of ready tasks seems to be promising. However, most of such heuristics consider only the DAG structure and/or task execution cost when making scheduling decision, while the data transmission cost, which receives particular attention in big data era, is often ignored. When the information about task execution time and data transmission time (even though might be inaccurate) is given, it might be interesting to investigate whether this information should be taken into account during scheduling, and if yes, how to. This paper presents an enhanced priority-based heuristic, which properly considers the DAG structure attributes, the task execution time and data transmission time estimates in the task prioritization. The evaluation result shows the proposed approach obtains better scheduling performance than existing solutions.