An evaluation of HPF compilers and the implementation of a parallel linear equation solver using HPF and MPI

Abstract

In this work, we evaluated the capabilities and performances of two commercially available HPF compilers, xlhpf[1] from IBM and pghpf[2] from the Portland Group. In particular, we examined the suitability of the two compilers for the development of a reservoir simulator. Because of the nature of reservoir simulation, multiple data distributions and data transfer between arrays of different data layouts are of great importance. An HPF compiler that does not provide these capabilities is unsuitable for the development of a parallel reservoir simulator. A detailed comparison of the functionalities of the two compilers and their suitabilities for reservoir simulator development are presented.To test the performance of the compilers, we used a parallel linear equation solver[3,4] that was developed to run on the CM5 and the Cray YMP. This solver is an important computational kernel in an in-house MPP reservoir simulator that is written in CM Fortran. Because the solver consumes about 90% of the CPU time in a typical reservoir simulation run with more than a million grid blocks, it can be used as an indicator of the performance of the simulator without having to convert it in its entirety. The solver was based on preconditioned Orthomin[5,6] and a truncated Neumann series preconditioner. Unlike solvers that are based on domain decomposition[7,8], the convergence behavior of this solver does not depend on the number of processors and therefore can scale better than domain decomposition based methods when the number of processors increases. This linear equation solver was found to be robust and efficient for the simulation of the giant oil reservoirs in Saudi Arabia and has been used in several field scale simulation studies with good success[4].The parallel solver was first implemented in HPF[9] and later in MPI[10] using hpf_local to improve the performance. Communication patterns of the solver include nearest neighbor communication, reduction function and general gather/scatter for handling the implicit bottom-hole pressure term. The solver has most of the features of a reservoir simulator to test the inter-processor communication bandwidth and floating point performance of the target machines. Its performance was compared for several MPP machines: CM5, IBM SP2, Cray T-3E and Cray Origin. The advantages and disadvantages of HPF and MPI are discussed from a programming as well as performance point of view.