Highly parallel structured adaptive mesh refinement using parallel language-based approaches

Abstract

Adaptive Mesh Refinement (AMR) calculations carried out on structured meshes play an exceedingly important role in several areas of science and engineering. This is so not just because AMR techniques allow us to carry out calculations very efficiently but also because they model very precisely the multi-scale fashion in which nature itself works. Many AMR applications are also amongst the most computationally intensive calculations undertaken making it necessary to use parallel supercomputers for their solution. While class library-based approaches are being attempted for parallel AMR we point out here that recent advances in the Fortran 90/95 standard and the OpenMP standard now make it possible to carry out highly parallel AMR calculations using language-based approaches. The language-based approaches offer several advantages over library-based approaches, the two principal ones being portability across parallel platforms and the best possible utilization of Distributed Shared Memory (DSM) hardware on machines that have such hardware. They also free up the applications scientist from being constrained by the static features of a class library. The choice of Fortran also ensures maximal reuse of pre-existing Fortran 77 applications and full Fortran 77-based processing efficiency on each computational node. Our implementation of the ideas presented here in the first author's RIEMANN framework essentially permits any serial, uniform grid, stencil-based Fortran code to be turned into a parallel AMR code. In this paper we first describe our strategy for using Fortran 90 in an object-oriented fashion. This permits AMR applications to be expressed in terms of familiar abstractions that are natural to the process of solving AMR hierarchies. We then describe the OpenMP features that are useful for parallel processing of AMR hierarchies in a load balanced fashion on multiprocessors. The automatic, parallel regridding of AMR hierarchies is also described. We then present a very efficient load balancer and show how it is to be used for load balanced solution of AMR hierarchies. Our load balancer is extremely general and should also see use in other disciplines. We follow this up with the application of the parallel AMR techniques developed here to the solution of elliptic and hyperbolic problems. For our elliptic problem we choose parallel, self-adaptive multigrid as an example. For our hyperbolic problem we choose time-dependent MHD as an example. In either case illustrative information is given about the adaptive processing of these systems. We also provide detailed scalability studies for both the above-mentioned problems which show that our methods scale extremely well up to several hundreds of processors.