Asynchronous modeling workflows in CyberWater with on-demand HPC/Cloud access

Abstract

Workflow management systems (WfMSs) are commonly used to organize/automate sequences of tasks as workflows to accelerate scientific discoveries. During complex workflow modeling, a local interactive workflow environment is desirable, as users usually rely on their rich, local environments for fast prototyping and refinements before they consider using more powerful computing resources. However, existing WfMSs do not simultaneously support local interactive workflow environments and HPC resources. In this paper, we present a mechanism for on-demand access to remote HPC resources from desktop/laptop-based workflow management software to compose, monitor, and analyze scientific workflows in the CyberWater project. CyberWater is an open-data and open-modeling software framework for environmental and water communities. In this work, we extend the open-model, open-data design of CyberWater with on-demand HPC accessing capacity. In particular, we design and implement the LaunchAgent and JobManager, which can be integrated into a local desktop/Laptop environment to allow on-demand usage of remote HPC resources for computational modeling workflows effectively and efficiently. LaunchAgent manages user authentication to remote resources, prepares computation-intensive or data-intensive tasks as batch jobs, submits jobs to remote resources, and monitors quality of services for users. LaunchAgent interacts seamlessly with other components in CyberWater, providing advantages of user-friendly feature-rich desktop software experience and increased computing power through on-demand HPC/Cloud usage. In our evaluations, we demonstrate how a hydrological modeling workflow that consists of both local and remote tasks can be constructed and show that our new on-demand HPC/Cloud usage helps speed up hydrology workflows while allowing asynchronous HPC/Cloud access from workflows using a desktop graphical user interface.