SwarmForm: A Distributed Workflow Management System with Task Clustering

Abstract

Scientific workflows are used to execute complex sets of interrelated calculations on High-Performance Computing environments. As workload management software do not support the execution of interrelated tasks, workflow management systems have been introduced to execute workflows on High-Performance Computing environments. Recent studies have introduced a new distributed architectural model that offers dynamic workflow execution capabilities to workflow management systems. The new architectural model executes workflows on a per-task basis. While this approach facilitates dynamic workflows, it adds a considerable overhead to workflows increasing their makespans substantially. As most workflows are static, task-wise execution of workflows degrades the performance of most workflows. In this paper, we introduce SwarmForm - an open source distributed workflow management system that introduces task clustering to the new architectural model. SwarmForm offers better performance than existing distributed workflow management systems by clustering workflow tasks to reduce overheads while allowing the users to choose between task-wise and cluster-wise execution of workflows depending on the nature of the workflow. SwarmForm enables the use of all the features introduced with the new architectural model while providing better makespans for scientific workflows.