A multiple-objective workflow scheduling framework for cloud data analytics

Abstract

One of the most important characteristics of a cloud system is elasticity in resources provisioning. Cloud fabric often composes of massive and heterogeneous types of resources allowing the sciences and engineering applications in many domains to collaboratively utilize the infrastructure. As the cloud systems are designed for a large number of users, a large volume of data, and various types of applications, efficient task management is needed for cloud data analytics. One of the popular methods used in task management is to represent a set of tasks with a workflow diagram, which can capture task decomposition, communication between subtasks, and cost of computation and communication. In this paper, we proposed a workflow scheduling framework that can efficiently schedule series workflows with multiple objectives onto a cloud system. Our designed framework uses a meta-heuristics method, called Artificial Bee Colony (ABC), to create an optimized scheduling plan. The framework allows multiple constraints and objectives to be set. Conflicts among objectives can also be resolved using Pareto-based technique. A series of experiments are then conducted to investigate the performance in comparison to the algorithms often used in cloud scheduling. Results show that our proposed method is able to reduce 57% cost and 50% scheduling time within a similar makespan of HEFT/LOSS for a typical scientific workflow like Chimera-2.