Distributed Memory Clusters Research Articles

Long-Time Simulations on High Resolution Meshes to Model Calcium Waves in a Heart Cell

A model for the flow of calcium on the scale of one heart cell is given by a system of time-dependent reaction-diffusion equations coupled by nonlinear reaction terms. Calcium ions enter into the cell at release units distributed throughout the cell and then diffuse. At each release unit, the probability for calcium to be released increases along with the concentration of calcium, thus creating a feedback loop of waves regenerating themselves repeatedly. The validation of this model requires simulations on the time scale of several repeated waves and on the spatial scale of the entire cell. This requires long-time studies on spatial meshes that need to have a high resolution to resolve the positions of the calcium release units throughout the entire cell. We detail the development of a special-purpose numerical method and parallel implementation for this problem. Parallel performance studies demonstrate the scalability of the implementation on a distributed-memory cluster with low-latency interconnect. Convergence studies verify convergence to analytical expectations and confirm the appropriateness of all numerical parameters. Application studies on the desired time and length scales confirm that the model exhibits the desired feedback mechanism for calcium currents through the release units at suitable high levels, but the long-time studies demonstrate also that the current model with its present parameters leads to excessive calcium concentrations over time. This phenomenon could only be observed using a computational method able to reach laboratory scale final times for a domain on the scale of a complete cell.

An introduction to parallel and vector scientific computing

In this text, students of applied mathematics, science and engineering are introduced to fundamental ways of thinking about the broad context of parallelism. The authors begin by giving the reader a deeper understanding of the issues through a general examination of timing, data dependencies, and communication. These ideas are implemented with respect to shared memory, parallel and vector processing, and distributed memory cluster computing. Threads, OpenMP, and MPI are covered, along with code examples in Fortran, C, and Java. The principles of parallel computation are applied throughout as the authors cover traditional topics in a first course in scientific computing. Building on the fundamentals of floating point representation and numerical error, a thorough treatment of numerical linear algebra and eigenvector/eigenvalue problems is provided. By studying how these algorithms parallelize, the reader is able to explore parallelism inherent in other computations, such as Monte Carlo methods.

Parallel implementation of a semidefinite programming solver based on CSDP on a distributed memory cluster

In this paper, we present the algorithmic framework and practical aspects of implementing a parallel version of a primal–dual semidefinite programming solver on a distributed memory computer cluster. Our implementation is based on the CSDP solver and uses a message passing interface and the ScaLAPACK library. A new feature is implemented to deal with problems that have rank-one constraint matrices. We show that significant improvement is obtained for a test set of problems with rank-one constraint matrices. Moreover, we show that very good parallel efficiency is obtained for large-scale problems where the number of linear equality constraints is very large compared to the block sizes of the positive semidefinite matrix variables.

Parallel Implementation of a Semidefinite Programming Solver Based on CSDP on a Distributed Memory Cluster

In this paper we present the algorithmic framework and practical aspects of implementing a parallel version of a primal-dual semidefinite programming solver on a distributed memory computer cluster. Our implementation is based on the CSDP solver and uses a message passing interface (MPI), and the ScaLAPACK library. A new feature is implemented to deal with problems that have rank-one constraint matrices. We show that significant improvement is obtained for a test set of problems with rank one constraint matrices. Moreover, we show that very good parallel efficiency is obtained for large-scale problems where the number of linear equality constraints is very large compared to the block sizes of the positive semidefinite matrix variables.

Slabpose Columnsort: A New Oblivious Algorithm for Out-of-Core Sorting on Distributed-Memory Clusters

Our goal is to develop a robust out-of-core sorting program for a distributed-memory cluster. The literature contains two dominant paradigms for out-of-core sorting algorithms: merging-based and partitioning-based. We explore a third paradigm, that of oblivious algorithms. Unlike the two dominant paradigms, oblivious algorithms do not depend on the input keys and therefore lead to predetermined I/O and communication patterns in an out-of-core setting. Predetermined I/O and communication patterns facilitate overlapping I/O, communication, and computation for efficient implementation. We have developed several out-of-core sorting programs using the paradigm of oblivious algorithms. Our baseline implementation, 3-pass columnsort, was based on Leighton's columnsort algorithm. Though efficient in terms of I/O and communication, 3-pass columnsort has a restriction on the maximum problem size. As our first effort toward relaxing this restriction, we developed two implementations: subblock columnsort and M-columnsort. Both of these implementations incur substantial performance costs: subblock columnsort performs additional disk I/O, and M-columnsort needs substantial amounts of extra communication and computation. In this paper we present slabpose columnsort, a new oblivious algorithm that we have designed explicitly for the out-of-core setting. Slabpose columnsort relaxes the problem-size restriction at no extra I/O or communication cost. Experimental evidence on a Beowulf cluster shows that unlike subblock columnsort and M-columnsort, slabpose columnsort runs almost as fast as 3-pass columnsort. To the best of our knowledge, our implementations are the first out-of-core multiprocessor sorting algorithms that make no assumptions about the keys and produce output that is perfectly load balanced and in the striped order assumed by the Parallel Disk Model.

A Galerkin Method for the Simulation of the Transient 2-D/2-D and 3-D/3-D Linear Boltzmann Equation

Many production steps used in the manufacturing of integrated circuits involve the deposition of material from the gas phase onto wafers. Models for these processes should account for gaseous transport in a range of flow regimes, from continuum flow to free molecular or Knudsen flow, and for chemical reactions at the wafer surface. We develop a kinetic transport and reaction model whose mathematical representation is a system of transient linear Boltzmann equations. In addition to time, a deterministic numerical solution of this system of kinetic equations requires the discretization of both position and velocity spaces, each two-dimensional for 2-D/2-D or each three-dimensional for 3-D/3-D simulations. Discretizing the velocity space by a spectral Galerkin method approximates each Boltzmann equation by a system of transient linear hyperbolic conservation laws. The classical choice of basis functions based on Hermite polynomials leads to dense coefficient matrices in this system. We use a collocation basis instead that directly yields diagonal coefficient matrices, allowing for more convenient simulations in higher dimensions. The systems of conservation laws are solved using the discontinuous Galerkin finite element method. First, we simulate chemical vapor deposition in both two and three dimensions in typical micron scale features as application example. Second, stability and convergence of the numerical method are demonstrated numerically in two and three dimensions. Third, we present parallel performance results which indicate that the implementation of the method possesses very good scalability on a distributed-memory cluster with a high-performance Myrinet interconnect.

A parallel FFT accelerated transient field-circuit simulator

A novel fast electromagnetic field-circuit simulator that permits the full-wave modeling of transients in nonlinear microwave circuits is proposed. This time-domain simulator is composed of two components: 1) a full-wave solver that models interactions of electromagnetic fields with conducting surfaces and finite dielectric volumes by solving time-domain surface and volume electric field integral equations, respectively, and 2) a circuit solver that models field interactions with lumped circuits, which are potentially active and nonlinear, by solving Kirchoff's equations through modified nodal analysis. These field and circuit analysis components are consistently interfaced and the resulting coupled set of nonlinear equations is evolved in time by a multidimensional Newton-Raphson scheme. The solution procedure is accelerated by allocating field- and circuit-related computations across the processors of a distributed-memory cluster, which communicate using the message-passing interface standard. Furthermore, the electromagnetic field solver, whose demand for computational resources far outpaces that of the circuit solver, is accelerated by a fast Fourier transform (FFT)-based algorithm, viz. the time-domain adaptive integral method. The resulting parallel FFT accelerated transient field-circuit simulator is applied to the analysis of various active and nonlinear microwave circuits, including power-combining arrays.

Parallel implementation of a monotone domain decomposition algorithm for nonlinear reaction-diffusion problems

Recently, a monotone iterative domain decomposition algorithm has been proposed for the numerical solution of nonlinear singularly perturbed reaction-diffusion problems. This paper describes a parallel implementation of the algorithm on a distributed memory cluster. Interprocess communication is effected by means of the MPI message passing library. For various domain decompositions, we give the convergence iteration count and execution time on up to 16 processors. The parallel scale-up of the algorithm improves as the number of mesh points is increased.

Load Balancing Parallel Explicit State Model Checking

This paper first identifies some of the key concerns about the techniques and algorithms developed for parallel model checking; specifically, the inherent problem with load balancing and large queue sizes resultant in a static partition algorithm. This paper then presents a load balancing algorithm to improve the run time performance in distributed model checking, reduce maximum queue size, and reduce the number of states expanded before error discovery. The load balancing algorithm is based on generalized dimension exchange (GDE). This paper presents an empirical analysis of the GDE based load balancing algorithm on three different supercomputing architectures—distributed memory clusters, Networks of Workstations (NOW) and shared memory machines. The analysis shows increased speedup, lower maximum queue sizes and fewer total states explored before error discovery on each of the architectures. Finally, this paper presents a study of the communication overhead incurred by using the load balancing algorithm, which although significant, does not offset performance gains.

Embedded Parallelization Approach for Optimization in Aerodynamic Design

A new efficient parallelization strategy for optimization of aerodynamic shapes is proposed. The optimization method employs a full Navier-Stokes solver for accurate estimation of the objective function. As such it requires huge computational resources which makes efficient parallelization crucial for successful promotion of the method to an engineering environment. The algorithm is based on a multilevel embedded parallelization approach, which includes (1) parallelization of the multiblock full Navier-Stokes solver with parallel CFD evaluation of objective function, (2) parallelization of optimization process with parallel optimal search on multiple search domains and, finally, (3) parallel grid generation. Applications (implemented on a 144-processors distributed memory cluster) include various transonic profile optimizations in the presence of nonlinear constraints. The results demonstrate that the approach combines high accuracy of optimization with high parallel efficiency. The proposed multilevel parallelization which efficiently makes use of computational power supplied by multiprocessor systems, leads to a significant computational time-saving and allows application of the method to practical aerodynamic design in the aircraft industry.

Evaluation of efficient security for BGP route announcements using parallel simulation

The Border Gateway Protocol (BGP) determines how Internet traffic is routed throughout the entire world; malicious behavior by one or more BGP speakers could create serious security issues. Since the protocol depends on a speaker honestly reporting path information sent by previous speakers and involves a large number of independent speakers, the Secure BGP (S-BGP) approach uses public-key cryptography to ensure that a malicious speaker cannot fabricate this information. However, such public-key cryptography is expensive: S-BGP requires a digital signature operation on each announcement sent to each peer, and a linear (in the length of the path) number of verifications on each receipt. We use simulation of AS models derived from the Internet to evaluate the impact that the processing costs of cryptography have on BGP convergence time. As the size of these models grows, inherent memory requirements grow beyond what is normally available in serial computers, motivating us to use distributed memory cluster computers, just to hold the model state. We find that under heavy load the convergence time using ordinary S-BGP is significantly larger than BGP. We examine the impact of highly aggressive caching and pre-computation optimizations for S-BGP, and find that convergence time is much closer to BGP. However, these optimizations may be unrealistic, and are certainly expensive of memory. We consequently use the structure of BGP processing to design optimizations that reduce cryptographic overhead by amortizing the cost of private-key signatures over many messages. We call this method Signature-Amortization (S-A). We find that S-A provides as good or better convergence times as the highly optimized S-BGP, but without the cost and complications of caching and pre-computation. These experiments––whose memory demands easily exceed 10Gb––are made possible using parallel simulation. They show that it is is possible therefore to minimize the impact route validation has on convergence, by being careful with signatures, rather than consumptive of memory.

Simulation of Sheared Suspensions With a Parallel Implementation of QDPD.

A parallel quaternion-based dissipative particle dynamics (QDPD) program has been developed in Fortran to study the flow properties of complex fluids subject to shear. The parallelization allows for simulations of greater size and complexity and is accomplished with a parallel link-cell spatial (domain) decomposition using MPI. The technique has novel features arising from the DPD formalism, the use of rigid body inclusions spread across processors, and a sheared boundary condition. A detailed discussion of our implementation is presented, along with results on two distributed memory architectures. A parallel speedup of 24.19 was obtained for a benchmark calculation on 27 processors of a distributed memory cluster.

Parallel Large Scale High Accuracy Navier-Stokes Computations on Distributed Memory Clusters

We present a highly scalable parallelization of a high-accuracy 3D serial multiblock Navier-Stokes solver. The code solves the full Navier-Stokes equations and is capable of performing large-scale computations for practical configurations in an industrial enviroment. The parallelization strategy is based on the geometrical domain decomposition principle, and on the overlapped communication and computation concept. The important advantage of the strategy is that the suggested type of message-passing ensures a very high scalability of the algorithm from the network point of view, because, on the average, the communication work per processor is not increased if the number of processors is increased. The parallel multiblock-structured Navier-Stokes solver based on the parallel virtual machine (PVM) routines was implemented on 106-processors distributed memory cluster managed by the MOSIX software package. Analysis of the results demonstrated a high level of parallel efficiency (speed up) of the computational algorithm. This allowed the reduction of the execution time for large-scale computations employing 10 million of grid points, from an estimated 46 days on the SGI ORIGIN 2000 computer (in the serial single-user mode) to 5–6 hours on 106-processors cluster. Thus, the parallel multiblock full Navier-Stokes code can be successfully used for large-scale practical aerodynamic simulations of a complete aircraft on millions-points grids on a daily basis, as needed in industry.

Partitioning strategies for parallel domain decomposition in modelling transport phenomena

Domain decomposition is a natural form of parallel preconditioning for discretised systems of partial differential equations, and the Additive Schwarz Method (ASM) is a convenient limiting form of domain decomposition preconditioning, since it maximises concurrency for a specified number of subdomains. However, except in the case of homogeneous, isotropic, elliptically-dominated systems, few of the many algorithmic parameters required by ASM possess a theoretical basis for their selection. This paper is an attempt to fill this gap experimentally. The parameters include the magnitude of anisotropic convective and diffusive effects, the order of the upwinded discretisation, the decomposition topology, decomposition orientation (relative to the anisotropy), the subdomain overlap, the accuracy of the subdomain solutions making up the composite preconditioner. The effects of these parameters on the numerical convergence rate, execution time, and parallel efficiency on distributed-memory parallel computers and workstations clusters are reported.

The Same-Source Parallel MM5

Beginning with the March 1998 release of the Penn State University/NCAR Mesoscale Model (MM5), and continuing through eight subsequent releases up to the present, the official version has run on distributed -memory (DM) parallel computers. Source translation and runtime library support minimize the impact of parallelization on the original model source code, with the result that the majority of code is line-for-line identical with the original version. Parallel performance and scaling are equivalent to earlier, hand-parallelized versions; the modifications have no effect when the code is compiled and run without the DM option. Supported computers include the IBM SP, Cray T3E, Fujitsu VPP, Compaq Alpha clusters, and clusters of PCs (so-called Beowulf clusters). The approach also is compatible with shared-memory parallel directives, allowing distributed-memory/shared-memory hybrid parallelization on distributed-memory clusters of symmetric multiprocessors.