On Scheduling of High-Throughput Scientific Workflows under Budget Constraints in Multi-Cloud Environments

Abstract

With the rapid development and deployment of cloud computing infrastructures, many applications in various scientific domains are increasingly utilizing cloud resources for big data storage and analysis. Particularly, it has become a significant challenge to manage and execute big data scientific workflows in multi-cloud environments to process streaming datasets. In this paper, within a three-layer workflow architecture with inter-and intra-cloud data transfer, we formulate a scientific workflow mapping problem under budget constraints to achieve the maximum throughput of streaming workflow applications. We propose a scheduling algorithm to identify the global bottleneck and maximize the throughput under budget constraints. Extensive simulation results show that the proposed algorithm exhibits superior performance over existing heuristic algorithms in scheduling streaming workflow applications.