SPMD OpenMP versus MPI for ocean models

Abstract

In two ocean models that use domain decomposition for scalability, communication between nodes is for the most part performed by calling an application specific API. This allows different low-level ‘message passing’ APIs to be added with relatively little effort. The first two supported APIs were MPI and the SHMEM put/get library. OpenMP can be used in Single Program Multiple Data (SPMD) mode by spawning N threads in the main program and having each thread act from then on similarly to a process in MPI. The initial port of one ocean model to SPMD OpenMP revealed several incompatibilities between thread-based and process-based SPMD coding styles. Adding support for threaded I/O was particularly painful, requiring modification to hundreds of lines of code. Several relatively minor additions to the OpenMP API were identified that would greatly simplify SMPD programming. Meanwhile, an alternative Fortran compiler-based SPMD API, Co-Array Fortran, became available on the Cray T3E. There is a simple mapping from SHMEM put/get library calls onto co-array assignment statements, so adding Co-Array Fortran support to the ocean models was straightforward. To extend Co-Array Fortran to machines other than the Cray T3E, a subset of the language is automatically translated into SPMD OpenMP via a nawk script. The performance of the ‘native’ OpenMP and translated Co-Array Fortran versions of the ocean model was virtually identical, so the former has been replaced by the latter (which is much easier to maintain). Copyright © 2000 John Wiley & Sons, Ltd.