Optimizing the Performance of Big Data Workflows in Multi-cloud Environments Under Budget Constraint

Abstract

Workflow techniques have been widely used as a major computing solution in many science domains. With the rapid deployment of cloud infrastructures around the globe and the economic benefit of cloud-based computing and storage services, an increasing number of scientific workflows have been shifted or are in active transition to clouds. As the scale of scientific applications continues to grow, it is now common to deploy data-and network-intensive computing workflows in multi-cloud environments, where inter-cloud data transfer oftentimes plays a significant role in both workflow performance and financial cost. We construct rigorous mathematical models to analyze the intra-and inter-cloud execution process of scientific workflows and formulate a budget-constrained workflow mapping problem to optimize the network performance of scientific workflows in multi-cloud environments. We show the proposed problem to be NP-complete and design a heuristic solution that takes into consideration module execution, data transfer, and I/O operations. The performance superiority of the proposed solution over existing methods is illustrated through extensive simulations.