Distributed Memory Computer Cluster Research Articles

Challenges and opportunities for the simulation of calcium waves on modern multi-core and many-core parallel computing platforms.

State-of-the-art distributed-memory computer clusters contain multicore CPUs with 16 and more cores. The second generation of the Intel Xeon Phi many-core processor has more than 60cores with 16GB of high-performance on-chip memory. We contrast the performance of the second-generation Intel Xeon Phi, code-named Knights Landing (KNL), with 68computational cores to the latest multicore CPU Intel Skylake with 18cores. A special-purpose code solving a system of nonlinear reaction-diffusion partial differential equations with several thousands of point sources modeled mathematically by Dirac delta distributions serves as realistic test bed. The system is discretized in space by the finite volume method and advanced by fully implicit time-stepping, with a matrix-free implementation that allows the complex model to have an extremely small memory footprint. The sample application is a seven variable model of calcium-induced calcium release (CICR) that models the interplay between electrical excitation, calcium signaling, and mechanical contraction in a heart cell. The results demonstrate that excellent parallel scalability is possible on both hardware platforms, but that modern multicore CPUs outperform the specialized many-core Intel Xeon Phi KNL architecture for a large class of problems such as systems of parabolic partial differential equations.

Scalable Hierarchical Parallel Algorithm for the Solution of Super Large-Scale Sparse Linear Equations

The parallel linear equations solver capable of effectively using 1000+ processors becomes the bottleneck of large-scale implicit engineering simulations. In this paper, we present a new hierarchical parallel master-slave-structural iterative algorithm for the solution of super large-scale sparse linear equations in a distributed memory computer cluster. Through alternatively performing global equilibrium computation and local relaxation, the specific accuracy requirement can be met in a few iterations. Moreover, each set/slave-processor majorly communicates with its nearest neighbors, and the transferring data between sets/slave-processors and the master-processor is always far below the communication between neighboring sets/slave-processors. The corresponding algorithm for implicit finite element analysis has been implemented based on the MPI library, and a super large 2-dimension square system of triangle-lattice truss structure under randomly distributed loadings is simulated with over 1 × 109 degrees of freedom (DOF) on up to 2001 processors of the “Exploration 100” cluster in Tsinghua University. The numerical experiments demonstrate that this algorithm has excellent parallel efficiency and high scalability, and it may have broad applications in other implicit simulations.

OpenCL‐based acceleration of the FDTD method in computational electromagnetics

SUMMARYThe evolution of processors into multi‐core architectures has led to the acceleration of scientific codes using numerous highly specialized processors, that is, multi‐core central processing units (CPUs), graphics processing units (GPUs) and also devices that merge both technologies in a single‐die chip. Development of parallel codes that are both scalable and portable between the processor architectures is challenging. To overcome this limitation, we investigated the acceleration of the finite‐difference time‐domain (FDTD) method in computational electromagnetics on modern computing architectures, that is, multi‐core CPUs and GPUs, through the use of Open Computing Language (OpenCL). Further extension of the OpenCL parallel programing model with the Message Passing Interface allows for the targeting of standard distributed memory computer clusters as well as clusters accelerated by GPUs. Portability between hardware manufactured by different vendors and highly specialized and parallel computing architectures is the main advantage of the developed FDTD solvers. The codes were coupled with a commercial simulation platform to evaluate the performance of the solvers in real‐world industrial scenarios. Although the portability resulted in a slightly reduced performance (10–35%) of the OpenCL‐accelerated FDTD simulations compared with the native Compute Unified Device Architecture or Open Multiprocessing implementations, the obtained benchmarking results of the OpenCL FDTD solvers on distributed memory systems show that the communication overhead can be hidden by computations for sufficiently large simulation domains with a scaling efficiency higher than 90%. Copyright © 2012 John Wiley & Sons, Ltd.

Scattering and absorption properties of polydisperse wavelength-sized particles covered with much smaller grains

Using the results of direct, numerically exact computer solutions of the Maxwell equations, we analyze scattering and absorption characteristics of polydisperse compound particles in the form of wavelength-sized spheres covered with a large number of much smaller spherical grains. The results pertain to the complex refractive indices 1.55+i0.0003, 1.55+i0.3, and 3+i0.1. We show that the optical effects of “dusting” wavelength-sized hosts by microscopic grains can vary depending on the number and size of the grains as well as on the complex refractive index. Our computations also demonstrate the high efficiency of the new superposition T-matrix code developed for use on distributed memory computer clusters.

Direct pore-level modeling of incompressible fluid flow in porous media

We present a dynamic particle-based model for direct pore-level modeling of incompressible viscous fluid flow in disordered porous media. The model is capable of simulating flow directly in three-dimensional high-resolution micro-CT images of rock samples. It is based on moving particle semi-implicit (MPS) method. We modify this technique in order to improve its stability for flow in porous media problems. Using the micro-CT image of a rock sample, the entire medium, i.e., solid and fluid, is discretized into particles. The incompressible Navier–Stokes equations are then solved for each particle using the MPS summations. The model handles highly irregular fluid–solid boundaries effectively. An algorithm to split and merge fluid particles is also introduced. To handle the computational load, we present a parallel version of the model that runs on distributed memory computer clusters. The accuracy of the model is validated against the analytical, numerical, and experimental data available in the literature. The validated model is then used to simulate both unsteady- and steady-state flow of an incompressible fluid directly in a representative elementary volume (REV) size micro-CT image of a naturally-occurring sandstone with 3.398 μm resolution. We analyze the quality and consistency of the predicted flow behavior and calculate absolute permeability using the steady-state flow rate.

Parallel implementation of the ADI‐FDTD method

AbstractWe present a parallel implementation of the three‐dimensional alternating direction implicit finite‐difference time‐domain (ADI‐FDTD) method in Cartesian coordinates using the message passing interface (MPI) library. Parallel implementations not only speed up computations but also increase the maximum solvable problem size. The application of the ADI scheme results in freeing the time‐step size from the Courant‐Friedrichs‐Lewy stability constraint. We demonstrate a speedup of the ADI‐FDTD method through parallelization, even though the communication overhead between the processors is proportional to the volume of the domain, rather than just the surface area of the subdomains as for standard parallel FDTD. We benchmarked our code on an IBM p690 symmetric multiprocessor and also a distributed memory computer cluster with a high‐bandwidth Infiniband interconnect. In both cases, satisfactory scalability and efficiency was demonstrated for simulation domains comprising up to 8 billion mesh cells. © 2009 Wiley Periodicals, Inc. Microwave Opt Technol Lett 51: 1298–1304, 2009; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/mop.24310

Parallelisation of Implicit Time Domain Methods: Progress with ADI-FDTD

The parallelisation of implicit time domain solvers is essential for applications requiring large, highly over-sampled meshes that exceed the memory limitations of standalone workstations (1). Prime application examples include large arrays of small antennas (2), and on- chip interconnects for state of the art integrated circuits (IC) (3). The need for highly over sampled meshes in these applications can be illustrated by example. Current on-chip IC interconnect widths can be as low as 22nm or less, yet cover an area > 100(mm) 2 and transmit signals with fundamental frequencies in the range DC-10GHz. Typically, transient efiects involving high frequencies are of the most interest, but even this can still require meshes with ¢x < (‚0=10 6 ), i.e., four to flve orders of magnitude smaller than for typical FDTD meshes (neglecting considerations relating to material properties). It is not practical to use explicit FDTD solvers for such meshes, even in parallel, because the Courant-Friedrichs-Lewy (CFL) stability criteria enforces a commensurate reduction in the time step size for numerical reasons. While implicit FDTD solvers, such as ADI-FDTD, ofier freedom from the CFL stability criteria, it has been presumed that parallel implementations on all but the most specialised architectures (4) would be of little beneflt due to the high communication overhead. However, we have been able to show that this is not the case (5). In this Paper we will present an overview of our work to date on the parallelisation of implicit time domain methods, including parallel ADI-FDTD (1,5) and parallel ADI-BOR-FDTD (6) on both symmetric multiprocessor (SMP) and distributed memory computer clusters (DMCC). We do not parallelise the tri-diagonal matrix solver itself, because each 2D matrix must have more than 4,000 (40,000) elements per direction before this becomes e-cient for SMP (DMCC) ma- chines. This requires 3D domains at are at or beyond current memory limits for state of the art machines. Instead, we employ domain decomposition and solve multiple, smaller, tri-diagonal ma- trix systems in parallel. Carefully organised data exchanges avoid unnecessary double-handling of data during communication, improving the parallel algorithm performance. We will describe our domain decomposition scheme, and show results for small and large domains, comparing par- allel FDTD and parallel ADI-FDTD and parallel BOR-FDTD with parallel ADI-BOR-FDTD. We demonstrate e-cient solutions of large full 3D meshes with 8 billion mesh cells for paral- lel ADI-FDTD. Since machines with SMP and DMCC architectures are widely available, our demonstration of parallel speed up represents an important step forward for the application of implicit time domain solvers for large, highly oversampled meshes with ¢x < 10 i2 ‚. We expect that our parallelisation approach can be adopted for related implicit FDTD methods.

Parallel implementation of a semidefinite programming solver based on CSDP on a distributed memory cluster

In this paper, we present the algorithmic framework and practical aspects of implementing a parallel version of a primal–dual semidefinite programming solver on a distributed memory computer cluster. Our implementation is based on the CSDP solver and uses a message passing interface and the ScaLAPACK library. A new feature is implemented to deal with problems that have rank-one constraint matrices. We show that significant improvement is obtained for a test set of problems with rank-one constraint matrices. Moreover, we show that very good parallel efficiency is obtained for large-scale problems where the number of linear equality constraints is very large compared to the block sizes of the positive semidefinite matrix variables.

Parallel Implementation of a Semidefinite Programming Solver Based on CSDP on a Distributed Memory Cluster

In this paper we present the algorithmic framework and practical aspects of implementing a parallel version of a primal-dual semidefinite programming solver on a distributed memory computer cluster. Our implementation is based on the CSDP solver and uses a message passing interface (MPI), and the ScaLAPACK library. A new feature is implemented to deal with problems that have rank-one constraint matrices. We show that significant improvement is obtained for a test set of problems with rank one constraint matrices. Moreover, we show that very good parallel efficiency is obtained for large-scale problems where the number of linear equality constraints is very large compared to the block sizes of the positive semidefinite matrix variables.

Time Domain Adaptive Integral Method for Surface Integral Equations

An efficient marching-on-in-time (MOT) scheme is presented for solving electric, magnetic, and combined field integral equations pertinent to the analysis of transient electromagnetic scattering from perfectly conducting surfaces residing in an unbounded homogenous medium. The proposed scheme is the extension of the frequency-domain adaptive integral/pre-corrected fast-Fourier transform (FFT) method to the time domain. Fields on the scatterer that are produced by space-time sources residing on its surface are computed: 1) by locally projecting, for each time step, all sources onto a uniform auxiliary grid that encases the scatterer; 2) by computing everywhere on this grid the transient fields produced by the resulting auxiliary sources via global, multilevel/blocked, space-time FFTs; 3) by locally interpolating these fields back onto the scatterer surface. As this procedure is inaccurate when source and observer points reside close to each other; and 4) near fields are computed classically, albeit (pre-)corrected, for errors introduced through the use of global FFTs. The proposed scheme has a computational complexity and memory requirement of O(N/sub t/N/sub s/log/sup 2/N/sub s/) and O(N/sub s//sup 3/2/) when applied to quasiplanar structures, and of O(N/sub t/N/sub s//sup 3/2/log/sup 2/N/sub s/) and O(N/sub s//sup 2/) when used to analyze scattering from general surfaces. Here, N/sub s/ and N/sub t/ denote the number of spatial and temporal degrees of freedom of the surface current density. These computational cost and memory requirements are contrasted to those of classical MOT solvers, which scale as O(N/sub t/N/sub s//sup 2/) and O(N/sub s//sup 2/), respectively. A parallel implementation of the scheme on a distributed-memory computer cluster that uses the message-passing interface is described. Simulation results demonstrate the accuracy, efficiency, and the parallel performance of the implementation.