A scalable parallel algorithm for direct-forcing immersed boundary method for multiphase flow simulation on spectral elements

Abstract

In this work, we propose a highly scalable parallel double binned ghost particle (DBGP) algorithm for direct-forcing immersed boundary spectral element method for multiphase flow simulations. In particular, the DBGP algorithm is designed to obtain fully distributed data storage and scalable data transfer across hundreds of thousands of processors. The proposed algorithm uses a queen and worker data structure for fully resolved particles to demarcate particle-level and marker-level quantities and communication. In the DBGP algorithm, each particle’s centroid is represented by a queen marker and the particle surface is covered with a uniform distribution of surface worker markers. The queen marker contains information on the translational and rotational motion of a particle and integrates the force and torque computed at all the worker markers, while the worker marker implements the fluid–particle interaction. Ghost queen and ghost worker markers are generated for each real queen and real worker marker during computation for particle-level and marker-level communications, respectively. A double Cartesian binning process is introduced that divides the physical domain into a coarse queen-bin and a fine worker-bin structure in three dimensions. The queen-bin and worker-bin sizes are determined by their zone of influence at the particle-level and marker-level communication, respectively. Bin-to-rank maps that relate each queen-bin and worker-bin to all the MPI ranks that they interact with are created. By using the queen/worker marker representation and two-layer bin-to-rank maps, data communication across very large number of MPI ranks is efficiently carried out. A scaling analysis has been conducted, showing excellent performance of the DBGP algorithm for up to 16,384 MPI ranks in both weak and strong scaling studies. The proposed method has been demonstrated to accurately predict sedimentation of particle clouds. The simulated correlation between the mean settling velocity and volume fraction is in good agreement with empirical correlations from previous studies.