Dynamic DAG scheduling for many-task computing of distributed eco-hydrological model

Abstract

The computing of distributed hydrological model at large scale is increasingly characterized by data intensive and computation intensive, especially for the multi-process coupling model. Parallel computing is one effective approach to cope with this situation. The easily extensible fine-grained parallelization method can substantially improve the computing efficiency. Based on many-task computing, we proposed a parallel scheme that the whole computing of the distributed hydrological model is split into tremendous amount of small sub-tasks which are directly dispatched into the cluster nodes by the traditional local resource managers (LRMs). The task-splitting method, the single task model and the representation of dependencies between tasks are also proposed. In order to efficiently schedule so many tasks, a dynamic DAG scheduling method based on critical path and depth is provided. The management of intermediate file, the control strategy for LRMs and the fault recovery is also introduced to deal with the problems encountered in the actual parallel implementation process. The parallel scheme is tested with an optimality-based distributed eco-hydrological model (disVOM) in the Poyang Lake sub-basin. It is demonstrated that our approach provide efficient computing performance.