Hybrid multi-grid parallelisation of WAVEWATCH III model on spherical multiple-cell grids

Abstract

Spherical Multiple-Cell (SMC) grid is an unstructured grid, supporting flexible domain shapes and multi-resolutions. It retains the quadrilateral cells as in the latitude-longitude (lat-lon) grid so that simple finite-difference schemes can be used. Sub-timesteps are applied on refined cells and grid cells are merged at high latitudes to relax the CFL restriction. A fixed reference direction is used in polar regions to solve vector polar problems. The SMC grid was implemented in the WAVEWATCH III (WW3) wave model in 2012 as an alternative for the lat-lon grid and updated in the latest WW3 V6.07. The WW3 model is parallelised by wave spectral component decomposition (CD) in MPI mode, which has a limit on number of MPI ranks. Hybrid or combined MPI-OpenMP parallelisation may extend the node usage but the OpenMP scalability flattens out beyond a few threads. Another parallelisation method that combines CD with domain decomposition (DD) is enabled in WW3 model by a multi-grid framework for further extension of node usage. Those regular lat-lon grid parallelisation options are gradually added to the SMC grid and this article reports the recent extension of the SMC grid into the multi-grid framework with hybrid parallelisation. The flexible domain shape of the SMC grid allows optimised domain splitting and minimised boundary exchanges. The combined CD-DD method is tested on SMC sub-grids with various hybrid node-thread combinations. Results indicate that switching from MPI to hybrid MPI-OpenMP mode can halve the global model elapsed time and using hybrid CD-DD on 3 SMC sub-grids may reduce it further by 30%. Elapsed time for one model day run is reduced from about 3 min on 12 nodes to less than 1 min on 90 or 180 nodes. Besides, the hybrid multi-grid method reduces memory demand on one computing node and allows future model updates for higher resolutions.