Fault-tolerant scheduling and data placement for scientific workflow processing in geo-distributed clouds

Abstract

Scientific workflow processing in geo-distributed cloud is crucial for the scheduling of large-scale tasks and the massive data placement among tasks. However, the task execution time and energy consumption of data transmission are two urgent issues when a scientific workflow is processed in the different geo-distributed data centers. Aiming at the data placement problem, this paper proposes a Lagrangian relaxation method. This method considers the workload balance, storage capacity, data dependency, transmission bandwidth, and transmission cost to obtain the minimum time of data transmission time. Aiming at the task scheduling problems, a fault-tolerant scheduling strategy is proposed. The strategy optimizes the task scheduling mechanism by considering the task execution time and energy consumption. Finally, the performance of the proposed methods is evaluated via extensive experiments In terms of the data placement, the experiment results imply that the data transmission time of the proposed relaxation algorithm can averagely achieve up to 14.61%, 38.03%, and 39.57% reduction of over that of ILP-FDP algorithm, GA-DPSO algorithm, and GPDP algorithm, respectively. As for the fault-tolerant scheduling, the energy consumption of the TSPT algorithm is the lowest. Compared with the MTS algorithm, EODS algorithm and EWTS algorithm, the average gains of the proposed algorithm are 15.33%, 16.65%, and 28.96%, respectively. Compared with the benchmark algorithms, the task execution time of the TSPT algorithm can averagely reduce up to 12.78,18.85 and 25.65, respectively.