A balancing act in heterogeneous computing – Developing the AWRA-Landscape data assimilation system

Abstract

The Australian Water Resource Assessment Landscape (AWRA-L) model is a landscape surface water model running at a daily time step on 277,770 grid cells. To make the best use of observations to guide the model results, an Ensemble Kalman filter (EnKF) technique is used to assimilate remotely sensed satellite data. It executes one hundred instances of the model in parallel, leading to substantial computational challenges by hydrologic modelling standards. The AWRA-L data assimilation system was originally implemented in the R language. While it is a concise implementation, runtime performance is not adequate to perform experiments at the scale of Australia, even using computational clusters. The AWRA-L data assimilation (DA) system must perform in exploratory research as well as in a broader operational model orchestrated by the software product Delft-FEWS (Flood Early Warning System) at the Bureau of Meteorology. The infrastructure available in each case is significantly different, as is the prioritisation of features. Software use in a research context requires execution speed as well as the possibility for the principal researcher to perform new experiments. These can be done on compute clusters, optionally with state of the art devices with Graphical Processing Units (GPU). The operational version of AWRA-L DA is designed to fit in a time stepping FEWS workflow. This workflow is such that AWRA-L DA must be run over the whole nation for a given time step before stepping temporally, a constraint not present in the research context. To reuse existing model code, the C# language is chosen for the core of AWRA-L DA, with a path to test and, if suitable, migrate to OpenCL for some of the computations. The operational system must scale well within a single process with multiple threads to maximize the use of the operational infrastructure. For the research needs it must also scale well across compute nodes, where network throughput is more likely to become a significant performance bottleneck. In order to assess the performance of computing on GPU devices for the AWRA-L DA, we use OpenCL.NET to run OpenCL code from the C# system. AWRA-L DA is added to an AWRA system where an implementation of the model already exists for calibration purposes. Calibration and DA contexts differ, notably in terms of primacy of the temporal versus spatial dimensions of the problem for computations. A custom implementation of AWRA-L for DA is necessary. We trial a code template technology to manage these parallel implementations, where the use of object-oriented and generic programming is technically not sufficient to limit duplications. The final assessment of the system in terms of runtime shows that the C# implementation scales well in a multi-threaded mode up to six to eight cores, enabling the efficient use of the resources of the operational system. A multi-process version dividing the problem spatially is used for research purposes, complemented with a final collation of partial results. Combining multi-threading and multiple processes in a cluster environment, we can balance between multi-threading overheads and the input/output (I/O) bottlenecks of a large number of processes. A usage with thirty two processes, each with two threads, is chosen for research experiments. We project that the use of a native language such as C++ would have brought an additional raw runtime performance of ~40%. The gain would however be less in a real use case due to I/O throughput considerations. Our trial of OpenCL, even limited to portions of the code already shows an improvement of 50% over a single threaded pure C# version. These figures suggest that were more runtime performance gains necessary for a problem of gridded nature such as AWRA-L DA, there are compelling reasons to consider GPU enabled computations.