Live monitoring, diagnostics and data management for the ACCESS suite of Earth-system simulations.

Abstract

As global Earth-System simulations rapidly increase in complexity as a direct result of available compute resources and advancing scientific model development, monitoring long-running models and managing associated big data outputs has become increasingly difficult. This has inspired a need for open community-focussed software tools that provide intuitive and efficient HPC-native functionality to accurately evaluate live model performance and behaviour, perform diagnostic scientific analyses, compare current results with alternative or reference models, and generate standardised data outputs. Addressing these challenges, we present a new community-driven open-source project Model Live Diagnostics (MLD) which integrates the ACCESS-NRI Intake Catalog API, together forming part of The Australian Earth-System Simulator (ACCESS-NRI) Model Evaluation and Diagnostics (MED) framework supporting climate science research within the Australian Community Climate and Earth System Simulator (ACCESS).   MLD is an open-source Python package optimised for use in JupyterLab sessions on the Australian NCI supercomputer Gadi. The primary purpose of MLD is to provide a simple, easy to use and accessible framework for the ACCESS modelling community to check, monitor, visualise and evaluate live running Earth-System model behaviour and progress. Utilising the ACCESS-NRI Intake Catalog API, MLD monitors a given model output data directory on Gadi, dynamically building an intake cataog of the most up-to-date data and automatically generating interactive plots to visualise model variables. From these data, MLD provides functions to perform light-weight diagnostic calculations, compare and plot results against reference models, and export data in standard Xarray format for integration into user workflows. MLD can be installed manually via Conda or PyPI, it comes pre-installed in existing Conda environments on Gadi, and the full source code is available on the project Github repository. MLD currently supports Earth System Model 1.5 (ACCESS-ESM1.5), Model for Ocean/Sea Ice (ACCESS-OM2) and Coupled Model 2 (ACCESS-CM2).