Embedding optimization in computational science workflows

Abstract

Workflows support the automation of scientific processes, providing mechanisms that underpin modern computational science. They facilitate access to remote instruments, databases and parallel and distributed computers. Importantly, they allow software pipelines that perform multiple complex simulations (leveraging distributed platforms), with one simulation driving another. Such an environment is ideal for computational science experiments that require the evaluation of a range of different scenarios “in silico” in an attempt to find ones that optimize a particular outcome. However, in general, existing workflow tools do not incorporate optimization algorithms, and thus whilst users can specify simulation pipelines, they need to invoke the workflow as a stand-alone computation within an external optimization tool. Moreover, many existing workflow engines do not leverage parallel and distributed computers, making them unsuitable for executing computational science simulations. To solve this problem, we have developed a methodology for integrating optimization algorithms directly into workflows. We implement a range of generic actors for an existing workflow system called Kepler, and discuss how they can be combined in flexible ways to support various different design strategies. We illustrate the system by applying it to an existing bio-engineering design problem running on a Grid of distributed clusters.