Dynamic MPI parallel task scheduling based on a master-worker pattern in cloud computing

Abstract

Load imbalance issues have become one of the main challenges in efficient task scheduling. On-demand computing resources that can be provided by the cloud infrastructure enable cost-efficiency for numerous application cases compared to on-premise resources that an organisation purchases and that might idle for non-peak situations. However, scheduling a large amount of tasks in parallel on the cloud nodes cannot always maintain the promised cost-efficiency due to the different workloads arising on these cloud nodes, caused by single point of failure, low bandwidth, and other unforeseen situations. Generated overhead and load imbalances between nodes lead to numerous paid resources lay idle. In our work, we propose a dynamic parallel task scheduling method by employing a master-worker model on a real-world engineering application executed on the Azure cloud. The main idea of our work is that we schedule tasks on cloud compute resources depending on the actual workload of each process instead of static-scheduled load.