Automated synchronization and boundary condition application for the Cactus Framework

Abstract

The Einstein Toolkit (ETK) is an open-source software platform for computational simulations in relativistic astrophysics and gravitational physics. The Cactus Framework (Cactus) comprises the core component of the ETK and handles creation of distributed data structures, parallelism, I/O, checkpointing, etc. The current Cactus scheduling system relies on the manual synchronization of ghost zones for grid functions (i.e. distributed matrices). Unfortunately, deciding which subroutines should synchronize which grid functions is a non-trivial problem. Incorrect synchronization can result in over synchronization (a performance problem) or under synchronization (an error). This issue also creates a barrier for new users and collaborators. We are developing a new scheduling and synchronization method for Cactus to improve the accessibility and efficiency of the ETK. Synchronization and application of boundary conditions are now handled automatically by Cactus. The new approach requires each subroutine to have and write declarations for individual grid functions. Before scheduled functions run, Cactus checks these declarations and performs any synchronization or boundary conditions as needed. This method removes the difficulty of deciding where synchronization should take place and removes unneeded synchronizations. The only knowledge required to provide these declarations is what grid functions a given subroutine uses and on which parts of the grid they are read or written. This new system is a first step in improving Cactus' scheduling methods to use more advanced and scalable techniques.