Dynamic load balancing with over decomposition in plasma plume simulations

Abstract

An electric propulsion plasma plume simulation employs the Particle-in-Cell (PIC) model for heavy species (i.e. ions and neutral atoms) and yields a highly non-uniform particle distribution at steady state. For parallel simulations, partitioning the computational domain uniformly and evenly assigning the subdomains to MPI processes would never lead to properly balanced loads across MPI processes. For this reason, a patch-based dynamic load balancing method with over-decomposition and Hilbert space-filling curve has been implemented into the Thermophysics Universal Research Framework (TURF). The data transfer is accomplished by utilizing TURF's serialization and deserialization feature, in which the tree hierarchical object structure is copied into a contiguous memory block of data, and the tree hierarchy is reconstructed after the data transfer. This generalized dynamic load balancing approach allows for the same routine to be used for any models besides the PIC model in TURF (e.g. fluid, continuum kinetic, hybrid models, etc). This paper first introduces individual pieces of TURF that enable dynamic load balancing when they are combined. Then, the dynamic load balancing method is demonstrated in three different particle simulations, showing performance gains when the number of subdomains is larger than that of MPI processes. The benefit is even more notable in simulations with highly non-uniform particle distributions at steady state, commonly seen in plasma plume simulations.