Asynchronous communication in spectral-element and discontinuous Galerkin methods for atmospheric dynamics – a case study using the High-Order Methods Modeling Environment (HOMME-homme_dg_branch)

Benjamin F Jamroz,Robert Klöfkorn

doi:10.5194/gmd-9-2881-2016

Abstract

Abstract. The scalability of computational applications on current and next-generation supercomputers is increasingly limited by the cost of inter-process communication. We implement non-blocking asynchronous communication in the High-Order Methods Modeling Environment for the time integration of the hydrostatic fluid equations using both the spectral-element and discontinuous Galerkin methods. This allows the overlap of computation with communication, effectively hiding some of the costs of communication. A novel detail about our approach is that it provides some data movement to be performed during the asynchronous communication even in the absence of other computations. This method produces significant performance and scalability gains in large-scale simulations.

Highlights

The Community Earth System Model (CESM) is a global climate model with full coupling between the atmosphere, ocean, land, sea-ice, and land-ice components (Gent et al, 2011)
In order to clearly explain the non-blocking asynchronous communication method, we first describe the data structures used in High-Order Methods Modeling Environment (HOMME) and the existing synchronous communication method. 2.1 Non-blocking communication Many high-performance scientific applications use Message Passing Interface (MPI) to communicate between processes in a distributed memory context
For all of the following runs we have used a cubed-sphere grid with ne elements along each edge of the cube for a total of E ≡ 6n2e total elements. 4.1 The Jablonowski–Williamson baroclinic wave instability test case The Jablonowski–Williamson baroclinic wave instability test case examines the evolution of an idealized baroclinic wave in the Northern Hemisphere

Summary

Introduction

The Community Earth System Model (CESM) is a global climate model with full coupling between the atmosphere, ocean, land, sea-ice, and land-ice components (Gent et al, 2011). HOMME was originally designed and optimized for the spectral-element (SE) method but nowadays supports a discontinuous Galerkin (DG) approach to advance the hydrostatic primitive equations Both methods have been chosen for their scalability on large distributed memory supercomputers. Unlike a finite-volume method where higher-order stencils have larger spatial extent, the SE and DG methods attain this property for arbitrary order, at the expense of a smaller time step These schemes limit the amount of inter-process communication, providing superior scalability in many applications. Point-to-point messaging is one of the communication paradigms implemented by MPI; others include reductions, broadcasts, scatters, and gathers This communication method is often used in the context of nearest-neighbor communication in the solution of partial differential equations using explicit time integration methods where data between neighboring grid elements (finite-volume cells, Galerkin elements) must be exchanged. Since blocking communication effectively causes a synchronization between the sending and receiving processes, this method is not widely used in high-performance parallel applications

Results

Discussion

Conclusion