Parallel Domain Decomposition Research Articles

A theoretical approach to domain decomposition for parallelization of Digital Terrain Analysis

Nowadays, many geocomputations are benefiting from parallel computing technologies. As a category of data and computationally intensive problems, Digital Terrain Analysis (DTA) can be improved by parallelization based on data parallelism. Load-balancing is one of the factors impacting the performance of a parallel DTA algorithm. In this article, we propose a theoretical approach that can schedule tasks for complex DTA parallel algorithms with respect to load-balancing among processors. First, we analyse the reason why the newly emerged theory of spatial computational domain is not suitable for complex DTA algorithms. Then, (1) by using the theory of cellular automata, a complex DTA algorithm can be modelled and thus decomposed into multiple simplex transitions; (2) by developing a trans-function for each simplex transition, load-balancing can be achieved for every transition, respectively; and (3) by solving the data reloading and results passing issues, all the simplex transitions can be linked back to the complex algorithm. The highlight of our approach is that the generated task scheduling solution always keeps adjusting itself as the algorithm proceeds one transition by another. This highlight also leads to a better load-balancing effect, which has been verified by a series of comparative experiments. In the experimental case, the parallel computing time using our approach becomes shorter than the one using the conventional approach. The decreasing ratio is 13.4%.

Application of PDSLin to the magnetic reconnection problem

Magnetic reconnection is a fundamental process in a magnetized plasma at both low and high magnetic Lundquist numbers (the ratio of the resistive diffusion time to the Alfven wave transit time), which occurs in a wide variety of laboratory and space plasmas, e.g. magnetic fusion experiments, the solar corona and the Earth's magnetotail. An implicit time advance for the two-fluid magnetic reconnection problem is known to be difficult because of the large condition number of the associated matrix. This is especially troublesome when the collisionless ion skin depth is large so that the Whistler waves, which cause the fast reconnection, dominate the physics (Yuan et al 2012 J. Comput. Phys. 231 5822–53). For small system sizes, a direct solver such as SuperLU can be employed to obtain an accurate solution as long as the condition number is bounded by the reciprocal of the floating-point machine precision. However, SuperLU scales effectively only to hundreds of processors or less. For larger system sizes, it has been shown that physics-based (Chacon and Knoll 2003 J. Comput. Phys. 188 573–92) or other preconditioners can be applied to provide adequate solver performance. In recent years, we have been developing a new algebraic hybrid linear solver, PDSLin (Parallel Domain decomposition Schur complement-based Linear solver) (Yamazaki and Li 2010 Proc. VECPAR pp 421–34 and Yamazaki et al 2011 Technical Report). In this work, we compare numerical results from a direct solver and the proposed hybrid solver for the magnetic reconnection problem and demonstrate that the new hybrid solver is scalable to thousands of processors while maintaining the same robustness as a direct solver.

Parallel numerical algorithm for simulation of a three-dimensional lid-driven cavity flow problem with the use of compact schemes

The parallel algorithm for simulation of the three-dimensional lid driven cavity flow problem with Reynolds numbers up to 10000 on a grid 128x128x128 was developed and implemented in the research. The problem is solved by numerical method where momentum equation is solved by a modified fractional step method with the use of compact scheme and pressure equation is solved by Fourier method in combination with the matrix sweep. The technique of the domain decomposition for parallelization of the proposed numerical method is described. In the process of computing and comparative experiment the dependency of the resulted acceleration and the coefficient of the calculation scalability on number of processors was found. Mathematics Subject Classication: 76F65

A Pressure-Stabilized Lagrange-Galerkin Method in a Parallel Domain Decomposition System

A pressure-stabilized Lagrange-Galerkin method is implemented in a parallel domain decomposition system in this work, and the new stabilization strategy is proved to be effective for large Reynolds number and Rayleigh number simulations. The symmetry of the stiffness matrix enables the interface problems of the linear system to be solved by the preconditioned conjugate method, and an incomplete balanced domain preconditioner is applied to the flow-thermal coupled problems. The methodology shows good parallel efficiency and high numerical scalability, and the new solver is validated by comparing with exact solutions and available benchmark results. It occupies less memory than classical product-type solvers; furthermore, it is capable of solving problems of over 30 million degrees of freedom within one day on a PC cluster of 80 cores.

Parallel algebraic domain decomposition solver for the solution of augmented systems

We consider the parallel iterative solution of indefinite linear systems given as augmented systems. Our numerical technique is based on an algebraic non-overlapping domain decomposition technique that only exploits the graph of the sparse matrix. This approach to high-performance, scalable solution of large sparse linear systems in parallel scientific computing, is to combine direct and iterative methods. We report numerical and parallel performance of the scheme on large matrices arising from the finite element discretization of linear elasticity in structural mechanics problems.

Parallel block interface domain decomposition methods for the 2D convection–diffusion equation

In this paper, a new block interface domain decomposition method (BI-DDM) with non-overlapping subdomains for the numerical solution of a two-dimensional convection–diffusion equation is presented. The block interface formulation is derived from the idea of using small groups of a certain number of mesh points where this group is treated explicitly similar to the way a single point is treated in the point method. The BI-DDM is incorporated with a correction phase which is able to economize further on the computing cost. The performance analysis of this method on several recently developed group iterative schemes implemented on a message-passing architecture are presented and discussed.

Parallel domain decomposition methods for beam-tracing

In this paper, a comparison of parallel beam-tracing methods is presented and an original parallel domain decomposition method is proposed to solve numerical acoustic problems. A hybrid method between the ray-tracing and the beam-tracing method is first introduced. Then, classical parallelization methods are exposed and compared on shared and distributed memory architectures. Finally, a new parallel method based on domain decomposition principles is proposed. This method allows to handling large-scale problems better than other existing methods when taking into account the input/output and preprocessing steps. Parallel numerical experiments, carried out on a real world problem -namely the acoustic pollution analysis within a large city-illustrate the performance of this new domain decomposition method.

Analysis of the time-Schwarz DDM on the heat PDE

We developed a parallel time domain decomposition method to solve systems of ODEs based on the Aitken–Schwarz domain decomposition method only in time (Linel P, Tromeur-Dervout D. Aitken–Schwarz and schur complement methods for time domain decomposition. In: Parallel computing: from multicores and GPU’s to Petascale. Advances in parallel computing, vol. 19; 2010. p. 75–82). The method transforms the initial time value problem into a time boundary values problem. This paper details the proof of the pure linear convergence of this DDM in case of linear system of ODEs and provides an optimized MPI parallel implementation when a regular size splitting of the time interval is performed. Then an extension of the method to the linear heat partial derivative equation is provided with a special care on boundary conditions to impose. A Fourier convergence analysis of this Schwarz DDM in time for this problem allows to limit the acceleration by Aitken’s technique to Fourier low modes only.

A Parallel Domain Decomposition Algorithm for Simulating Blood Flow with Incompressible Navier-Stokes Equations with Resistive Boundary Condition

AbstractWe introduce and study a parallel domain decomposition algorithm for the simulation of blood flow in compliant arteries using a fully-coupled system of nonlinear partial differential equations consisting of a linear elasticity equation and the incompressible Navier-Stokes equations with a resistive outflow boundary condition. The system is discretized with a finite element method on unstructured moving meshes and solved by a Newton-Krylov algorithm preconditioned with an overlapping restricted additive Schwarz method. The resistive outflow boundary condition plays an interesting role in the accuracy of the blood flow simulation and we provide a numerical comparison of its accuracy with the standard pressure type boundary condition. We also discuss the parallel performance of the implicit domain decomposition method for solving the fully coupled nonlinear system on a supercomputer with a few hundred processors.

Numerical modeling of ventilation process and thermal status of the noise-heat protective housing of a gas-turbine plant on parallel computers

We study the thermal status of a noise-heat protective housing (NHPH), where a gas turbine plant (GTP) is located. The analysis of thermal conditions requires consideration of heat transfer caused by radiation emitted by the heated elements of the housing. In this paper, a mathematical model is proposed to calculate the rate and temperature of the air flow inside the NHPH, as well as the temperature of the NHPH wall-material, making allowance for radiation heat transfer. A parallel algorithm is designed for solving the cluster gas dynamic problem. The optimization problem is solved by applying the parallel domain decomposition method. Calculations are made for the NHPH of one of the available of gas-compressor units. The results confirm that the heat transfer caused by radiation should be taken into account in the solution of a solid-state thermal conductivity problem for NHPH walls. There is good agreement between the calculated temperatures and the temperatures measured at certain points on the external surface of NHPH walls.

A Frozen Jacobian Multiscale Mortar Preconditioner for Nonlinear Interface Operators

We present an efficient approach for preconditioning systems arising in multiphase flow in a parallel domain decomposition framework known as the mortar mixed finite element method. Subdomains are coupled together with appropriate interface conditions using mortar finite elements. These conditions are enforced using an inexact Newton--Krylov method, which traditionally required the solution of nonlinear subdomain problems on each interface iteration. A new preconditioner is formed by constructing a multiscale basis on each subdomain for a fixed Jacobian and time step. This basis contains the solutions of nonlinear subdomain problems for each degree of freedom in the mortar space and is applied using an efficient linear combination. Numerical experiments demonstrate the relative computational savings of recomputing the multiscale preconditioner sparingly throughout the simulation versus the traditional approach.

A New Parallel Domain-Decomposed Chebyshev Collocation Method for Atmospheric and Oceanic Modeling

Spectral methods seek the solution to a differential equation in terms of series of known smooth function. The Chebyshev series possesses the exponential-convergence property regardless of the imposed boundary condition, and therefore is suited for the regional modeling. We propose a new domain-decomposed Chebyshev collocation method which facilitates an efficient parallel implementation. The boundary conditions for the individual sub-domains are exchanged through one grid interval overlapping. This approach is validated using the one dimensional advection equation and the inviscid Burgers' equation. We further tested the vortex formation and propagation problems using two-dimensional nonlinear shallow water equations. The domain decomposition approach in general gave more accurate solutions compared to that of the single domain calculation. Moreover, our approach retains the exponential error convergence and conservation of mass and the quadratic quantities such as kinetic energy and enstrophy. The efficiency of our method is greater than one and increases with the number of processors, with the optimal speed up of 29 and efficiency 3.7 in 8 processors. Efficiency greater than one was obtained due to the reduction the degrees of freedom in each sub-domain that reduces the spectral operational count and also due to a larger time step allowed in the sub-domain method. The communication overhead begins to dominate when the number of processors further increases, but the method still results in an efficiency of 0.9 in 16 processors. As a result, the parallel domain-decomposition Chebyshev method may serve as an efficient alternative for atmospheric and oceanic modeling.

GPU implementations of the bond fluctuation model

We present two parallel implementations of the bond fluctuation model on graphics processors that outperform by a factor of up to 50 times an equivalent implementation on single CPU processor. The first algorithm is a parallelized version of an accelerated MC method published earlier in [S. Nedelcu, J.-U. Sommer, Single chain dynamics in polymer networks: a Monte Carlo study, J. Chem. Phys. 130 (2009) 204902]. In this first algorithm we use the parallel domain decomposition technique to avoid monomer collisions. In contrast, in the second algorithm we associate each monomer with a parallel process, where all monomers in the system are attempted to move simultaneously. In both cases, only monomer moves that result in allowed bonds and preserve lattice occupancy are accepted. To validate the correctness of the GPU algorithms we simulated monodisperse polymer melts at monomer number density 0.5 and compared static and dynamical properties with standard CPU implementations. We found good agreement between the CPU and the GPU results, which demonstrates the equivalence of the serial and parallel implementations. The influence of higher monomer number density is discussed.

Hybrid and Parallel Domain-Decomposition Methods Development to Enable Monte Carlo for Reactor Analyses

** This paper describes code and methods development at the Oak Ridge National Laboratory focused on enabling high-fidelity, large-scale reactor analyses with Monte Carlo (MC). Current state-of-the-art tools and methods used to perform “real” commercial reactor analyses have several undesirable features, the most significant of which is the non-rigorous spatial decomposition scheme. Monte Carlo methods, which allow detailed and accurate modeling of the full geometry and are considered the “gold standard” for radiation transport solutions, are playing an ever-increasing role in correcting and/or verifying the deterministic, multi-level spatial decomposition methodology in current practice. However, the prohibitive computational requirements associated with obtaining fully converged, system-wide solutions restrict the role of MC to benchmarking deterministic results at a limited number of state-points for a limited number of relevant quantities. The goal of this research is to change this paradigm by enabling direct use of MC for full-core reactor analyses. The most significant of the many technical challenges that must be overcome are the slow, non-uniform convergence of system-wide MC estimates and the memory requirements associated with detailed solutions throughout a reactor (problems involving hundreds of millions of different material and tally regions due to fuel irradiation, temperature distributions, and the needs associated with multi-physics code coupling). To address these challenges, our research has focused on the development and implementation of (1) a novel hybrid deterministic/MC method for determining high-precision fluxes throughout the problem space in k-eigenvalue problems and (2) an efficient MC domain-decomposition (DD) algorithm that partitions the problem phase space onto multiple processors for massively parallel systems, with statistical uncertainty estimation. The hybrid method development is based on an extension of the FW-CADIS method, which attempts to achieve uniform statistical uncertainty throughout a designated problem space. The MC DD development is being implemented in conjunction with the Denovo deterministic radiation transport package to have direct access to the 3-D, massively parallel discrete-ordinates solver (to support the hybrid method) and the associated parallel routines and structure. This paper describes the hybrid method, its implementation, and initial testing results for a realistic 2-D quarter core pressurized-water reactor model and also describes the MC DD algorithm and its implementation.

Three‐dimensional parallel frequency‐domain visco‐acoustic wave modelling based on a hybrid direct/iterative solver

ABSTRACTWe present a parallel domain decomposition method based on a hybrid direct‐iterative solver for 3D frequency‐domain modelling of visco‐acoustic waves. The method is developed as a modelling engine for frequency‐domain full waveform inversion. Frequency‐domain seismic modelling reduces to the solution of a large and sparse system of linear equations, resulting from the discretization of the heterogeneous Helmholtz equation. Our approach to the high‐performance, scalable solution of large sparse linear systems is to combine direct and iterative methods. Such a hybrid approach exploits the advantages of both direct and iterative methods. The iterative component uses a small amount of memory and provides a natural way for parallelization. The direct part has favourable numerical properties for multiple right‐hand side modelling. The domain decomposition is based upon the algebraic Schur complement method, which allows for the iterative solution of a reduced system, the solution of which is the wavefield at the interfaces between the subdomains. Once the interface unknowns have been computed, the wavefield at the interior of each subdomain is efficiently computed by local substitutions. The reduced Schur complement system is solved with the generalized minimum residual method and is preconditioned by an algebraic additive Schwarz preconditioner. A direct solver is used to factorize local impedance matrices defined on each subdomain. Theoretical analysis shows that the time complexity of the hybrid solver is the same as that of iterative solver and time‐domain approaches for single frequency modelling. Simulations are performed in the SEG/EAGE overthrust and the salt models for frequencies up to 12.5 Hz. The number of iterations increases linearly with the number of subdomains for a given computational domain but the elapsed time of the iterative resolution remains almost constant. The number of iterations also increases linearly with frequencies, when the grid interval is adapted to the frequencies and the size of the subdomains is kept constant over frequency. These results make the cost of the hybrid solver of the same order as that of finite‐difference time‐domain modeling for one‐frequency modelling. Although the hybrid approach allows one to tackle larger problems than the direct‐solver approach, further improvements are needed to mitigate the computational burden of the iterative component in the context of multisource modelling. On the numerical side, the use of block iterative solvers and of incremental two‐level deflating preconditioners and on the parallel implementation side, the use of two levels of parallelism in the domain decomposition method should mitigate this computational burden.

A new electromagnetic particle-in-cell model with adaptive mesh refinement for high-performance parallel computation

A new electromagnetic particle-in-cell (EMPIC) model with adaptive mesh refinement (AMR) has been developed to achieve high-performance parallel computation in distributed memory system. For minimizing the amount and frequency of inter-processor communications, the present study uses the staggering grid scheme with the charge conservation method, which consists only of the local operations. However, the scheme provides no numerical damping for electromagnetic waves regardless of the wavenumber, which results in significant noise in the refinement region that eventually covers over physical signals. In order to suppress the electromagnetic noise, the present study introduces a smoothing method which gives numerical damping preferentially for short wavelength modes. The test simulations show that only a weak smoothing results in drastic reduction in the noise, so that the implementation of the AMR is possible in the staggering grid scheme. The computational load balance among the processors is maintained by a new method termed the adaptive block technique for the domain decomposition parallelization. The adaptive block technique controls the subdomain (block) structure dynamically associated with the system evolution, such that all the blocks have almost the same number of particles. The performance of the present code is evaluated for the simulations of the current sheet evolution. The test simulations demonstrate that the usage of the adaptive block technique as well as the staggering grid scheme enhances significantly the parallel efficiency of the AMR-EMPIC model.

Quasi-real-time simulation of rotating drum using discrete element method with parallel GPU computing

Real-time simulation of industrial equipment is a huge challenge nowadays. The high performance and fine-grained parallel computing provided by graphics processing units (GPUs) bring us closer to our goals. In this article, an industrial-scale rotating drum is simulated using simplified discrete element method (DEM) without consideration of the tangential components of contact force and particle rotation. A single GPU is used first to simulate a small model system with about 8000 particles in real-time, and the simulation is then scaled up to industrial scale using more than 200 GPUs in a 1D domain-decomposition parallelization mode. The overall speed is about 1/11 of the real-time. Optimization of the communication part of the parallel GPU codes can speed up the simulation further, indicating that such real-time simulations have not only methodological but also industrial implications in the near future.

A two-level parallel algorithm for material nonlinearity problems

An efficient two-level domain decomposition parallel algorithm is suggested to solve large-DOF structural problems with nonlinear material models generating unsymmetric tangent matrices, such as a group of plastic-damage material models. The parallel version of the stabilized bi-conjugate gradient method is developed to solve unsymmetric coarse problems iteratively. In the present approach the coarse DOF system is solved parallelly on each processor rather than the whole system equation to minimize the data communication between processors, which is appropriate to maintain the computing performance on a non-supercomputer level cluster system. The performance test results show that the suggested algorithm provides scalability on computing performance and an efficient approach to solve large-DOF nonlinear structural problems on a cluster system.

Parallel Galerkin domain decomposition procedures based on the streamline diffusion method for convection–diffusion problems

Based upon the streamline diffusion method, parallel Galerkin domain decomposition procedures for convection–diffusion problems are given. These procedures use implicit method in the sub-domains and simple explicit flux calculations on the inter-boundaries of sub-domains by integral mean method or extrapolation method to predict the inner-boundary conditions. Thus, the parallelism can be achieved by these procedures. The explicit nature of the flux calculations induces a time step limitation that is necessary to preserve stability. Artificial diffusion parameters δ are given. By analysis, optimal order error estimate is derived in a norm which is stronger than L 2 -norm for these procedures. This error estimate not only includes the optimal H 1 -norm error estimate, but also includes the error estimate along the streamline direction ‖ β ⋅ ∇ ( u − U ) ‖ , which cannot be achieved by standard finite element method. Experimental results are presented to confirm theoretical results.

Parallel Domain-decomposed Taiwan Multi-scale Community Ocean Model (PD-TIMCOM)

The Parallel Domain-decomposed Taiwan Multi-scale Community Ocean Model (PD-TIMCOM) was developed to provide a flexible and efficient community ocean model for simulating a variety of idealized and real ocean flows over a wide range of scales and boundary conditions. The model is particularly targeted at resolving multi-scale dynamics in the ocean environment, ranging from small scale turbulence to the global circulation gyres. The novel parallel algorithm improves the efficiency of the Error Vector Propagating (EVP) method, a simple direct solver for the typical pressure Poisson equations in the PD-TIMCOM. The new approach is ideal for multiple processes and takes advantage of parallel domain-decomposition, which can significantly reduce the operational counts and computational costs simultaneously. The speed-up is proportional to the number of domains, thus making the PD-TIMCOM a practical eddy-resolving global ocean model for climate projection. We illustrate the parallel performance based on the 1/4° global adaptation of PD-TIMCOM. Our results show accurate meso-scale variability, the reasonable separation of several western boundary currents from the coast, and the appropriate watermass distribution in the global ocean. Consistent with satellite altimetry, the results also show clear mean fronts in the Kuroshio Extension and extensive Kuroshio–Oyashio interaction. This leads to a quasi-equilibrium eddy field associated with three meandering jets in the Kuroshio Extension and Gulf Stream.