Scientific Workflow Execution Using a Dynamic Runtime Model

Abstract

Research projects consist of several kinds of steps covering, e.g., individual procedures to gather data, as well as different ways to process and analyze it. Moreover, the individual steps of the project consist of a sequence of tasks that together form a workflow. With the continuous advancements in computer science, scientists have more and more access to different kinds of infrastructures and tools which are suited for different types of experiments. Cloud computing is one of the premier infrastructures to perform experiments on, as it provides flexible, on-demand computing resources that are off-premise. Still, a uniform and platform-independent orchestration of these resources remains challenging, especially when various infrastructures and human interactions are required throughout the execution of a scientific workflow. This thesis provides an approach to allow scientists to define infrastructural resources for individual tasks within their workflows and dynamically shift them throughout the workflow execution. To reach this goal, we couple recent advancements in cloud orchestration with runtime models and open cloud standards. Combined, we aim for highly tailored workflows while fostering the reuse of already existing methodologies built around the standards. To realize this objective, we build a cloud runtime model orchestration process based on the Open Cloud Computing Interface (OCCI) standard. We extend the OCCI data model with workflow elements and corresponding capabilities to model cloud deployments for individual workflow tasks. This allows forming a runtime workflow model that can be coupled to different systems such as production clouds or simulation environments. To demonstrate the feasibility of the approach, we perform several experiments to validate the standard conform orchestration process and assess the applicability of the runtime workflow model coupled to cloud infrastructures. Our studies show that runtime models are a suitable knowledge base for adaptive behavior including the modeling and runtime representation of highly tailored workflows. We observe that the orchestration of cloud deployments and the execution of workflows follow a reoccurring pattern which can be described via a sequence of runtime states. Especially the uniform interface of OCCI allows for an automatic management and reuse of existing systems and standards. By monitoring and reflecting operational properties, the runtime model fosters decision-making processes not only for self-adaptive systems but also for human users. We show that the runtime model enables human-in-the-loop activities, allowing, e.g., to influence the control flow or the parallelization of workflow tasks at runtime. Furthermore, the runtime model can be attached to different environments allowing to test adaptation and workflow behavior.