Three-dimensional Domain Decomposition Research Articles

Efficient parallel strategy for molecular plasmonics – A numerical tool for integrating Maxwell-Schrödinger equations in three dimensions

An efficient parallelization approach to simulate optical properties of ensembles of quantum emitters in realistic electromagnetic environments is considered. It relies on balancing computing load of utilized processors and is built into three-dimensional domain decomposition methodology implemented for numerical integration of Maxwell's equations. The approach employed enables directly accessing dynamics of collective effects as the number of molecules in simulations can be drastically increased. Numerical experiments measuring speedup factors demonstrate the efficiency of the proposed methodology. As an example, we consider dynamics of nearly 700,000 diatomic molecules with ro-vibrational degrees of freedom explicitly accounted for coupled to electromagnetic radiation crafted by periodic arrays of split-ring resonators and triangular nanoholes. As an application of the approach, dissociation dynamics under strong coupling conditions is scrutinized. It is demonstrated that the dissociation rates are significantly affected near polaritonic frequencies.

An upright bottomless vertical cylinder with baffles floating in waves

Damping of the surge and pitch motions, as well as the first lateral sloshing mode in a rigid free-floating upright circular dock with bilge boxes and open bottom is investigated. Model tests are carried out on a 0.80 m diameter model in regular waves with wave periods near the highest natural sloshing period, and the internal free-surface elevation and model’s rigid body motions are measured. Perforated and solid annular baffles of relatively small widths are also installed inside the dock at various submergences. The experimental results are compared to a semi-analytical approach, where a three-dimensional domain decomposition method based on linear potential flow theory is adopted to calculate the hydrodynamic coefficients and exciting forces in heave, surge and pitch. A reduced natural sloshing frequency, as well as a damping ratio estimated from the energy dissipated due to flow separation from the baffles, are introduced in the free-surface boundary conditions to model the effects of the baffle. It shows good agreement with experimental data when the ratio between the draft of the baffle and the internal radius of the cylinder is dB/a=0.27, and tends to under-predict the damping ratio for shallower drafts, most likely due to free-surface interactions. The solid baffle damps the sloshing response most efficiently, reducing the amplitude at the resonant peak by more than 56%.

Motion of a floating body in a harbour by domain decomposition method

A three-dimensional domain decomposition method is used to solve the problem of wave interaction with a ship floating inside a harbour with arbitrary shape. The linearized velocity potential theory is adopted. The total fluid domain is divided into two sub-ones: one for the harbour and the other for the external open sea. Boundary integral equations together with the free surface Green function are used in the both domains. Matching conditions are imposed on the interface of the two sub-domains to ensure the velocity and pressure continuity. The advantage of the domain decomposition method over the single domain method is that it removes the coastal surface from the boundary integral equation. This subsequently removes the need for elements on the coastal wall when the equation is discretized. The accuracy of the method is demonstrated through convergence study and through the comparison with the published data. Extensive results through the hydrodynamic coefficients, wave exciting forces and ship motions are provided. Highly oscillatory behaviour is observed and its mechanism is discussed. Finally, the effects of incident wave direction, ship location as well as the harbour topography are investigated in detail.

A dual communicator and dual grid-resolution algorithm for petascale simulations of turbulent mixing at high Schmidt number

A new dual-communicator algorithm with very favorable performance characteristics has been developed for direct numerical simulation (DNS) of turbulent mixing of a passive scalar governed by an advection–diffusion equation. We focus on the regime of high Schmidt number (Sc), where because of low molecular diffusivity the grid-resolution requirements for the scalar field are stricter than those for the velocity field by a factor Sc. Computational throughput is improved by simulating the velocity field on a coarse grid of Nv3 points with a Fourier pseudo-spectral (FPS) method, while the passive scalar is simulated on a fine grid of Nθ3 points with a combined compact finite difference (CCD) scheme which computes first and second derivatives at eighth-order accuracy. A static three-dimensional domain decomposition and a parallel solution algorithm for the CCD scheme are used to avoid the heavy communication cost of memory transposes. A kernel is used to evaluate several approaches to optimize the performance of the CCD routines, which account for 60% of the overall simulation cost. On the petascale supercomputer Blue Waters at the University of Illinois, Urbana–Champaign, scalability is improved substantially with a hybrid MPI-OpenMP approach in which a dedicated thread per NUMA domain overlaps communication calls with computational tasks performed by a separate team of threads spawned using OpenMP nested parallelism. At a target production problem size of 81923 (0.5 trillion) grid points on 262,144 cores, CCD timings are reduced by 34% compared to a pure-MPI implementation. Timings for 163843 (4 trillion) grid points on 524,288 cores encouragingly maintain scalability greater than 90%, although the wall clock time is too high for production runs at this size. Performance monitoring with CrayPat for problem sizes up to 40963 shows that the CCD routines can achieve nearly 6% of the peak flop rate. The new DNS code is built upon two existing FPS and CCD codes. With the grid ratio Nθ∕Nv=8, the disparity in the computational requirements for the velocity and scalar problems is addressed by splitting the global communicator MPI_COMM_WORLD into disjoint communicators for the velocity and scalar fields, respectively. Inter-communicator transfer of the velocity field from the velocity communicator to the scalar communicator is handled with discrete send and non-blocking receive calls, which are overlapped with other operations on the scalar communicator. For production simulations at Nθ=8192 and Nv=1024 on 262,144 cores for the scalar field, the DNS code achieves 94% strong scaling relative to 65,536 cores and 92% weak scaling relative to Nθ=1024 and Nv=128 on 512 cores.

Efficient parallelization of analytic bond-order potentials for large-scale atomistic simulations

Analytic bond-order potentials (BOPs) provide a way to compute atomistic properties with controllable accuracy. For large-scale computations of heterogeneous compounds at the atomistic level, both the computational efficiency and memory demand of BOP implementations have to be optimized. Since the evaluation of BOPs is a local operation within a finite environment, the parallelization concepts known from short-range interacting particle simulations can be applied to improve the performance of these simulations. In this work, several efficient parallelization methods for BOPs that use three-dimensional domain decomposition schemes are described. The schemes are implemented into the bond-order potential code BOPfox, and their performance is measured in a series of benchmarks. Systems of up to several millions of atoms are simulated on a high performance computing system, and parallel scaling is demonstrated for up to thousands of processors.

Direct pore-to-core up-scaling of displacement processes: Dynamic pore network modeling and experimentation

Summary We present a new dynamic pore network model that is capable of up-scaling two-phase flow processes from pore to core. This dynamic model provides a platform to study various flow processes in porous media at the core scale using the pore-scale physics. The most critical features of this platform include (1) the incorporation of viscous, capillary, and gravity pressure drops in pore-scale displacement thresholds, (2) wetting-phase corner flow in capillary elements with angular cross-sections, (3) adjustments of corner interfaces between wetting and non-wetting phases based on changes in local capillary pressure, (4) simultaneous injection of wetting and non-wetting phases from the inlet of the medium at constant flow rates that makes the study of steady-state processes possible, (5) heavy parallelization using a three-dimensional domain decomposition scheme that enables the study of two-phase flow at the core scale, and (6) constant pressure boundary condition at the outlet. For the validation of the dynamic model, three two-phase miniature core-flooding experiments were performed in a state-of-the-art micro core-flooding system integrated with a high-resolution X-ray micro-CT scanner. The dynamic model was rigorously validated by comparing the predicted local saturation profiles, fractional flow curves, relative permeabilities, and residual oil saturations against their experimental counterparts. The validated dynamic model was then used to study low-IFT and high-viscosity two-phase flow processes and investigate the effect of high capillary number on relative permeabilities and residual oil saturation.

A three-dimensional domain decomposition method for large-scale DFT electronic structure calculations

With tens of petaflops supercomputers already in operation and exaflops machines expected to appear within the next 10 years, efficient parallel computational methods are required to take advantage of such extreme-scale machines. In this paper, we present a three-dimensional domain decomposition scheme for enabling large-scale electronic structure calculations based on density functional theory (DFT) on massively parallel computers. It is composed of two methods: (i) the atom decomposition method and (ii) the grid decomposition method. In the former method, we develop a modified recursive bisection method based on the moment of inertia tensor to reorder the atoms along a principal axis so that atoms that are close in real space are also close on the axis to ensure data locality. The atoms are then divided into sub-domains depending on their projections onto the principal axis in a balanced way among the processes. In the latter method, we define four data structures for the partitioning of grid points that are carefully constructed to make data locality consistent with that of the clustered atoms for minimizing data communications between the processes. We also propose a decomposition method for solving the Poisson equation using the three-dimensional FFT in Hartree potential calculation, which is shown to be better in terms of communication efficiency than a previously proposed parallelization method based on a two-dimensional decomposition. For evaluation, we perform benchmark calculations with our open-source DFT code, OpenMX, paying particular attention to the O(N) Krylov subspace method. The results show that our scheme exhibits good strong and weak scaling properties, with the parallel efficiency at 131,072 cores being 67.7% compared to the baseline of 16,384 cores with 131,072 atoms of the diamond structure on the K computer.

Performance measurement of magnetohydrodynamic code for space plasma on typical scalar-type supercomputer systems with a large number of cores

The computational performance of magnetohydrodynamic (MHD) code is evaluated on two typical scalar-type supercomputer systems. We have carried out performance tuning of a three-dimensional MHD code for space plasma simulations on the HA8000 (with 8192 cores) and SR16000/L2 (with 1344 cores) supercomputer systems. For parallelization of the MHD code, we use four different methods, that is, regular one-dimensional, two-dimensional and three-dimensional domain decomposition methods and a cache-hit type of three-dimensional domain decomposition method. We found that the regular three-dimensional decomposition of the MHD model is suitable for the HA8000 system, and the cache-hit type of three-dimensional decomposition is suitable for SR16000/L2 systems. As a result of these runs, we achieved a performance efficiency of almost 15% on the HA8000 and 20% on the SR16000/L2 for MHD code.

Accelerating large-scale simulation of seismic wave propagation by multi-GPUs and three-dimensional domain decomposition

We adopted the GPU (graphics processing unit) to accelerate the large-scale finite-difference simulation of seismic wave propagation. The simulation can benefit from the high-memory bandwidth of GPU because it is a “memory intensive” problem. In a single-GPU case we achieved a performance of about 56 GFlops, which was about 45-fold faster than that achieved by a single core of the host central processing unit (CPU). We confirmed that the optimized use of fast shared memory and registers were essential for performance. In the multi-GPU case with three-dimensional domain decomposition, the non-contiguous memory alignment in the ghost zones was found to impose quite long time in data transfer between GPU and the host node. This problem was solved by using contiguous memory buffers for ghost zones. We achieved a performance of about 2.2 TFlops by using 120 GPUs and 330 GB of total memory: nearly (or more than) 2200 cores of host CPUs would be required to achieve the same performance. The weak scaling was nearly proportional to the number of GPUs. We therefore conclude that GPU computing for large-scale simulation of seismic wave propagation is a promising approach as a faster simulation is possible with reduced computational resources compared to CPUs.