Abstract
In this paper, we present a set of best practices for workflow design and implementation for numerical weather prediction models and meteorological data service, which have been in operation in China Meteorological Administration (CMA) for years and have been proven effective in reliably managing the complexities of large-scale meteorological related workflows. Based on the previous work on the platforms, we argue that a minimum set of guidelines including workflow scheme, module design, implementation standards and maintenance consideration during the whole establishment of the platform are highly recommended, serving to reduce the need for future maintenance and adjustment. A significant gain in performance can be achieved through the workflow-based projects. We believe that a good workflow system plays an important role in the weather forecast service, providing a useful tool for monitoring the whole process, fixing the errors, repairing a workflow, or redesigning an equivalent workflow pattern with new components.
Highlights
China Meteorological Administration (CMA) has been utilizing high performance computing systems (HPC) since the 1980s
We present a set of best practices for workflow design and implementation for numerical weather prediction models and meteorological data service, which have been in operation in China Meteorological Administration (CMA) for years and have been proven effective in reliably managing the complexities of large-scale meteorological related workflows
All jobs must be submitted to Loadleveler via llsubmit. It is at the ecFLow or submission script level where certain environment specific variables must be set. ecFlow executes tasks and receives acknowledgements from the associated job when it changes status or when it sends events. It does this using child commands embedded in the scripts. ecFlow stores the relationship between tasks and is able to submit tasks dependent on triggers. ecFlow is complemented by ecflow_ui, its graphical interface that allows users to have immediate knowledge, using color coding, of the status of the various programs or processes handled by the scheduler (Figure 2)
Summary
China Meteorological Administration (CMA) has been utilizing high performance computing systems (HPC) since the 1980s. The current HPC system was introduced in 2013, with a peak performance of 1054.2TFlops. There are two identical subsystems, one for production, and another for research (Figure 1) [1]. The computing cluster consists of 560 nodes, which is interconnected with the storage cluster via Infiniband network. There are login nodes which are used for users to log on the system and submit the jobs. Those service nodes are used for managing the jobs dispatch and workload balance. CMA has a unique set of challenges since we require highly available, accurate, monitorable, and flexible systems, while simultaneously supporting the business and critical research development efforts
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Free) 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
