KubeAdaptor: A docking framework for workflow containerization on Kubernetes

Abstract

As Kubernetes becomes the infrastructure of the cloud-native era, the integration of workflow systems with Kubernetes is gaining more and more popularity. However, when migrating the workflow systems to the Kubernetes platform, the inconsistency between the workflow scheduling algorithms and Kubernetes in task scheduling order seriously reduces the execution efficiency of the workflow systems. Besides, the integration of existing container-based workflow systems with Kubernetes lacks a general docking framework and requires various built-in tools and specialized technique supports, which brings high research costs and inconvenience to users. To this end, this paper proposes a cloud-native workflow engine named KubeAdaptor, a general docking framework to integrate workflow systems with Kubernetes and implement workflow containerization on Kubernetes, aiming to ensure a consistent task scheduling order between workflow scheduling algorithms in workflow systems and Kubernetes. We present the KubeAdaptor architecture and elaborate on the functionality implementation, fault tolerance management, and the event trigger mechanism within the KubeAdaptor. Experimental results on four real-world scientific workflows show that the KubeAdaptor as a docking framework ensures the consistency of the workflow scheduling algorithms and Kubernetes in the task scheduling order. Compared with the state-of-the-art Argo workflow engine, the KubeAdaptor achieves better performance in terms of the average execution time of the task pod, average workflow lifecycle, and resource usage rate.