Terascale Optimal PDE Simulations (TOPS) Center

Abstract

This report covers the period from Oct. 2002 to Sep. 2004 when Old Dominion University (ODU) was the lead institution for the TOPS ISIC, until in Oct. 2004 Columbia University replaced ODU as the lead institution. The TOPS members from ODU focused on various aspects of the linear and nonlinear solver infrastructure required by the partial differential equations simulation codes, working directly with SciDAC teams from the Fusion Energy Sciences program: the Center for Extended agnetohydrodynamic Modeling (CEMM) at Princeton, and with the Center for Magnetic Reconnection Studies (CMRS) at University of New Hampshire. With CEMM we worked with their MHD simulation code, called M3D, which is semi-implicit, requiring linear solves but no onlinear solves. We contributed several improvements to their current semi-implicit code. Among these was the use of multilevel reconditioning, which provides optimal scaling. This was done through the multigrid preconditioner available in Hypre, another major solver package available in TOPS. We also provided them direct solver functionality for their linear solves since they may be required for more accurate solutions in some regimes. With the CMRS group, we implemented a fully implicit parallel magnetic reconnection simulation code, built on top of PETSc. Our first attempt was a Krylov linear iteration (GMRES because of the lack of symmetry), within each nonlinear (Newton) iteration, with optimal multilevel preconditioning, using the geometric multigrid preconditioner from PETSc. However, for reasons that we have not yet fully understood, the multigrid preconditioner fails early in the simulation, breaking the outer Newton iteration. Much better results were obtained after switching from optimal multilevel preconditioning to suboptimal one level preconditioning. Our current code, based on the additive Schwartz preconditioner from in PETSc, with ILU on subdomains, scales reasonably well, while matching the output of the original explicit code. The new Newton-Krylov-Schwarz implicitcode can take time-steps that are hundreds or thousands of times larger than the explicit code. During the three year period of this grant, we published thirteen papers and gave several invited talks at international conferences. Work on these TOPS projects continues with Columbia University as lead until Sep. 2006.