Distributed Memory Computer Systems Research Articles

Massively parallelized interpolated factored Green function method

This paper presents the first parallel implementation of the novel “Interpolated Factored Green Function” (IFGF) method introduced recently for the accelerated evaluation of discrete integral operators arising in wave scattering and other areas (Bauinger and Bruno, Jour. Computat. Phys., 2021). On the basis of the hierarchical IFGF interpolation strategy, the proposed (hybrid MPI-OpenMP) parallel implementation results in efficient data communication, and it scales up to large numbers of cores—without any hard limitations on the number of cores efficiently employed. Moreover, on any given number of cores, the proposed parallel approach preserves the O(Nlog⁡N) computing cost inherent in the sequential version of the IFGF algorithm. Unlike other approaches, the IFGF method does not utilize the Fast Fourier Transform (FFT), and it is thus better suited for efficient parallelization in distributed-memory computer systems. In particular, the IFGF method relies on a “peer-to-peer” strategy wherein, at every level, field propagation is directly enacted via “exchanges” between “peer” polynomials of constant degree, without data accumulation in large-scale “telephone-central” mathematical constructs such as those used in the Fast Multipole Method (FMM) and pure FFT-based approaches. A variety of numerical results presented in this paper illustrate the character of the proposed parallel algorithm, in particular demonstrating scaling from 1 to all 1,680 cores available in the High Performance Computing cluster used, and for problems of up to 4,096 wavelengths in acoustic size.

Large-scale earthquake sequence simulations on 3-D non-planar faults using the boundary element method accelerated by lattice H-matrices

SUMMARY Large-scale earthquake sequence simulations using the boundary element method (BEM) incur extreme computational costs through multiplying a dense matrix with a slip rate vector. Hierarchical matrices (H-matrices) have often been used to accelerate this multiplication. However, the complexity of the structures of the H-matrices and the communication costs between processors limit their scalability, and they therefore cannot be used efficiently in distributed memory computer systems. Lattice H-matrices have recently been proposed as a tool to improve the parallel scalability of H-matrices. In this study, we developed a method for earthquake sequence simulations applicable to 3-D non-planar faults with lattice H-matrices. We present a simulation example and verify the mesh convergence of our method for a 3-D non-planar thrust fault using rectangular and triangular discretizations. We also performed performance and scalability analyses of our code. Our simulations, using over ${10}^5$ degrees of freedom, demonstrated a parallel acceleration beyond ${10}^4$ MPI processors and a > 10-fold acceleration over the best performance when the normal H-matrices are used. Using this code, we can perform unprecedented large-scale earthquake sequence simulations on geometrically complex faults with supercomputers. The software is made an open-source and freely available.

“Interpolated Factored Green Function” method for accelerated solution of scattering problems

This paper presents a novel Interpolated Factored Green Function method (IFGF) for the accelerated evaluation of the integral operators in scattering theory and other areas. Like existing acceleration methods in these fields, the IFGF algorithm evaluates the action of Green function-based integral operators at a cost of O(Nlog⁡N) operations for an N-point surface mesh. The IFGF strategy, which leads to an extremely simple algorithm, capitalizes on slow variations inherent in a certain Green function analytic factor, which is analytic up to and including infinity, and which therefore allows for accelerated evaluation of fields produced by groups of sources on the basis of a recursive application of classical interpolation methods. Unlike other approaches, the IFGF method does not utilize the Fast Fourier Transform (FFT), and is thus better suited than other methods for efficient parallelization in distributed-memory computer systems. Only a serial implementation of the algorithm is considered in this paper, however, whose efficiency in terms of memory and speed is illustrated by means of a variety of numerical experiments—including a 43 min., single-core operator evaluation (on 10 GB of peak memory), with a relative error of 1.5×10−2, for a problem of acoustic size of 512 wavelengths.

Sparse, iterative simulation methods for one-dimensional laminar flames

Sparse, iterative simulation methods for one-dimensional laminar flames are proposed. The resulting steady and unsteady flame solvers exploit approximate Jacobians to greatly reduce the computational cost associated with matrix operations. The constant, non-unity Lewis number assumption is introduced to further reduce the computational cost. The solvers are also parallelized to reduce time-to-solution on distributed memory computer systems. Computed laminar flame speeds and species profiles for a range of chemical mechanisms (from 10 to 2878 species) are compared against a well-validated commercial code and are found to be consistent within solver tolerances. The computation times of both the unsteady and steady solutions increase only linearly with the number of species, which is a significant improvement over the quadratic or cubic scaling of existing steady-state flame solvers. For the largest mechanism tested, the steady-state flame solver is two orders of magnitude faster than commonly-used codes. The use of an approximate Jacobian is shown to reduce the rate of convergence for the steady-state solver, but does not significantly affect the domain of convergence. The steady-state solver with approximate Jacobian is thus well suited for computationally efficient laminar flame speed sweeps with large kinetic mechanisms.

Parameter estimation of qualitative biological regulatory networks on high performance computing hardware

BackgroundBiological Regulatory Networks (BRNs) are responsible for developmental and maintenance related functions in organisms. These functions are implemented by the dynamics of BRNs and are sensitive to regulations enforced by specific activators and inhibitors. The logical modeling formalism by René Thomas incorporates this sensitivity with a set of logical parameters modulated by available regulators, varying with time. With the increase in complexity of BRNs in terms of number of entities and their interactions, the task of parameters estimation becomes computationally expensive with existing sequential SMBioNET tool. We extend the existing sequential implementation of SMBioNET by using a data decomposition approach using a Java messaging library called MPJ Express. The approach divides the parameters space into different regions and each region is then explored in parallel on High Performance Computing (HPC) hardware.ResultsThe performance of the parallel approach is evaluated on BRNs of different sizes, and experimental results on multicore and cluster computers showed almost linear speed-up. This parallel code can be executed on a wide range of concurrent hardware including laptops equipped with multicore processors, and specialized distributed memory computer systems. To demonstrate the application of parallel implementation, we selected a case study of Hexosamine Biosynthetic Pathway (HBP) in cancer progression to identify potential therapeutic targets against cancer. A set of logical parameters were computed for HBP model that directs the biological system to a state of recovery. Furthermore, the parameters also suggest a potential therapeutic intervention that restores homeostasis. Additionally, the performance of parallel application was also evaluated on a network (comprising of 23 entities) of Fibroblast Growth Factor Signalling in Drosophila melanogaster.ConclusionsQualitative modeling framework is widely used for investigating dynamics of biological regulatory networks. However, computation of model parameters in qualitative modeling is computationally intensive. In this work, we presented results of our Java based parallel implementation that provides almost linear speed-up on both multicore and cluster platforms. The parallel implementation is available at https://psmbionet.github.io.

Iterative load-balancing method with multigrid level relaxation for particle simulation with short-range interactions

We developed dynamic load-balancing algorithms for Particle Simulation Methods (PSM) involving short-range interactions, such as Smoothed Particle Hydrodynamics (SPH), Moving Particle Semi-implicit method (MPS), and Discrete Element method (DEM). These are needed to handle billions of particles modeled in large distributed-memory computer systems. Our method utilizes flexible orthogonal domain decomposition, allowing the sub-domain boundaries in the column to be different for each row. The imbalances in the execution time between parallel logical processes are treated as a nonlinear residual. Load-balancing is achieved by minimizing the residual within the framework of an iterative nonlinear solver, combined with a multigrid technique in the local smoother. Our iterative method is suitable for adjusting the sub-domain frequently by monitoring the performance of each computational process because it is computationally cheaper in terms of communication and memory costs than non-iterative methods. Numerical tests demonstrated the ability of our approach to handle workload imbalances arising from a non-uniform particle distribution, differences in particle types, or heterogeneous computer architecture which was difficult with previously proposed methods. We analyzed the parallel efficiency and scalability of our method using Earth simulator and K-computer supercomputer systems.

Parallel Algorithms Design for the Motion Simulation of Tethered Satellite Systems Using Graph Models

Tethered satellite systems (TSS) are characterized by uneven distribution of mass and environment parameters in space. As a result the use of mathematical models with distributed parameters is required. This kind of systems is described by the differential equations with complex boundary conditions. The complexity of the boundary conditions is caused by the presence of the end-bodies that perform spatial fluctuations, and by the variable length of the tether. Computer simulation of TSS motion takes a long time. This paper presents a parallel algorithm for motion simulation of the TSS and representation of this algorithm in the form of a graph model in graph-symbolic programming technology. The speedup of a parallel program that implements the proposed algorithm is evaluated for shared memory and distributed memory computer systems. This evaluation is compared with the results of experiments made on computer cluster. The advantages of using graph models of algorithms for modeling the motion of the TSS are discussed.

PSelInv—A Distributed Memory Parallel Algorithm for Selected Inversion

We describe an efficient parallel implementation of the selected inversion algorithm for distributed memory computer systems, which we call PSelInv. The PSelInv method computes selected elements of a general sparse matrix A that can be decomposed as A = LU , where L is lower triangular and U is upper triangular. The implementation described in this article focuses on the case of sparse symmetric matrices. It contains an interface that is compatible with the distributed memory parallel sparse direct factorization SuperLU_DIST. However, the underlying data structure and design of PSelInv allows it to be easily combined with other factorization routines, such as PARDISO. We discuss general parallelization strategies such as data and task distribution schemes. In particular, we describe how to exploit the concurrency exposed by the elimination tree associated with the LU factorization of A . We demonstrate the efficiency and accuracy of PSelInv by presenting several numerical experiments. In particular, we show that PSelInv can run efficiently on more than 4,000 cores for a modestly sized matrix. We also demonstrate how PSelInv can be used to accelerate large-scale electronic structure calculations.

Solving PDEs with radial basis functions

Finite differences provided the first numerical approach that permitted large-scale simulations in many applications areas, such as geophysical fluid dynamics. As accuracy and integration time requirements gradually increased, the focus shifted from finite differences to a variety of different spectral methods. During the last few years, radial basis functions, in particular in their ‘local’ RBF-FD form, have taken the major step from being mostly a curiosity approach for small-scale PDE ‘toy problems’ to becoming a major contender also for very large simulations on advanced distributed memory computer systems. Being entirely mesh-free, RBF-FD discretizations are also particularly easy to implement, even when local refinements are needed. This article gives some background to this development, and highlights some recent results.

Методы параллельного решения СЛАУ на системах с распределенной памятью в библиотеке Krylov

The paper presents an approach to creation of black-box parallel iterative solver, which is used in Krylov library for solving systems of linear algebraic equations (SLAEs) with large sparse matrices in CSR format arising from discretization of multidimensional boundary value problems. A variant of one-dimensional algebraic decomposition method is offered. The algorithm is based on breadth-first search of SLAE’s adjacency graph that allows to reduce the matrix to block-tridiagonal form. The algebraic solver is based on additive Schwarz method which naturally suits distributed memory computer systems. The generalized minimal residual method is used to solve the SLAEs arising from relations on subdomains’ boundaries. Auxiliary subdomain systems are solved with Intel MKL’s multithreaded direct solver PARDISO. Implemented algorithms were tested on the numerical solution of the series of computational mathematics problems, such as problems of hydrodynamics, diffusion-convection equations, problems of electromagnetism and others. Adduced numerical experiments results show the effectiveness of the presented algorithms for multiprocessor computational systems with distributed memory.

The reordered BiCGStab method for distributed memory computer systems

A new reordered formulation of the preconditioned BiCGStab iterative method for the system of linear equations with large sparse nonsymmetric matrix is presented. The algorithm is reformulated in order to improve the efficiency on distributed memory computer systems. It allows to avoid all global synchronization points of the inner product operations. The order of computations permits to overlap the communication time of the inner products by the preconditioning computations. The efficiency of the implemented method with the algebraic multigrid preconditioner is demonstrated by the scalability results for the MPI and the hybrid (MPI +SHM) programming models.

Comparison Study of Performance of Parallel Steady State Solver on Different Computer Architectures

A parallel solver for steady state analysis of nonlinear circuits is presented. The solver uses the Message Passing Interface specification for communications and is suitable for steady state simulation of very large nonlinear circuits. Performance of the solver is investigated on symmetric multiprocessing, non-uniform memory access, and distributed memory computer systems. Impact of memory subsystem constraints on the solver efficiency is evaluated.

Parallel unstructured mesh generation by an advancing front method

Mesh generation is a critical step in high fidelity computational simulations. High-quality and high-density meshes are required to accurately capture the complex physical phenomena. A robust approach for a parallel framework has been developed to generate large-scale meshes in a short period of time. A coarse tetrahedral mesh is generated first to provide the basis of block interfaces and then is partitioned into a number of sub-domains using METIS partitioning algorithms. A volume mesh is generated on each sub-domain in parallel using an advancing front method. Dynamic load balancing is achieved by evenly distributing work among the processors. All the sub-domains are combined to create a single volume mesh. The combined volume mesh can be smoothed to remove the artifacts in the interfaces between sub-domains. A void region is defined inside each sub-domain to reduce the data points during the smoothing operation. The scalability of the parallel mesh generation is evaluated to quantify the improvement on shared- and distributed-memory computer systems.

Study of OpenMP applications on the InfiniBand-based software distributed shared-memory system

For the past decades computer engineers have focused on building high-performance and large-scale computer systems with low-cost. One of the examples is a distributed-memory computer system like a cluster, where fast processing nodes to use commodity processors are connected through a high speed network. But it is not easy to develop applications on this system, because a programmer must consider all data and control dependences between processes and program them explicitly. For alleviating this problem the distributed virtual shared-memory (DVSM) system has been proposed. It is well known that the performance of the DVSM system highly depends on the network’s performance and programming semantics, and currently its performance is very limited on a conventional network. Recently many advanced hardware-based interconnection technologies have been introduced, and one of them is the InfiniBand Architecture (IBA) which supports shared-memory programming semantics by means of remote direct-memory access (RDMA) and atomic operations. In this paper, we present the implementation of our InfiniBand-based DVSM system and analyze the performance of SPEC OMP benchmarks in detail by comparing with the DVSM based on the traditional network architecture and the hardware shared-memory multiprocessor (SMP) system. As experiment result, we show that our DVSM system to use full features of the IBA can improve the performance significantly over the IPoIB-based traditional system on the IBA, and furthermore the performance of one application on the IBA-based DVSM system is better than on the hardware SMP.

Parallel iterative solution of large‐scale acoustic scattering problems using exact non reflecting conditions on distributed memory computer systems

Parallel iterative methods for fast solution of large‐scale acoustic radiation and scattering problems are developed using exact Dirichlet‐to‐Neumann (DtN) nonreflecting boundaries. For elongated scatterers such as submarines, it is shown that the generalization of the DtN to elliptical/spheroidal artificial boundaries improves significantly the computational efficiency of accurate finite element methods for the solution of acoustic scattering problems. The outer‐product structure of the DtN map is exploited as a low‐rank update of the system matrix to efficiently compute the matrix‐by‐vector products found in Krylov subspace based iterative methods. For the complex non‐Hermitian matrices resulting from the Helmholtz equation, a distributed‐memory parallel BICG‐STAB iterative method is used in conjunction with a hybrid parallel SSOR/Jacobi preconditioner. The domain decomposition with interface minimization was performed to ensure optimal inter‐processor communication. For the distributed memory architectures tested, including Linux/Intel Beowulf clusters, when implemented as a low‐rank update, the nonlocal character of the DtN map shows little impact on the scale up or parallel efficiency compared to approximate local boundary conditions. [Work supported by NSF.]

Parallel processing for boundary element computations on distributed systems

A parallel boundary integral algorithm for solving boundary value problems on distributed memory computer systems is presented in this paper. The paper focuses on parallelizations of the influence coefficients matrix generation and the solution of the resulting linear system, which are the two main parts of boundary integral formulations. The distributed parallel boundary integral algorithm presented in this paper generates a part of the influence coefficients matrix on each computation node of a multicomputer platform and stores that part in the local memory. The distributed influence coefficients matrix is then used in its partitioned form by the parallel linear system solver to obtain the solution. The matrix of coefficients is large, dense and nonsymmetric. Three parallel linear system solvers are presented. Among them, Algorithm-1 and Algorithm-2 are based on the conjugate-gradient-squared (CGS) method, Algorithm-3 is based on a direct method. It is found that distributed memory parallel algorithms are very much machine dependent. The performance of a parallel linear system solver depends not only on the method of solution but also on the multicomputer network topology, and its ratio of node computation speed to network data transfer bandwidth. By selecting the algorithm which fits the characteristics of a distributed computer system hardware one can achieve relative maximum performance. The performance characteristics of distributed linear system solvers are studied in order to select an optimum method for the available hardware at minimum development cost. The performance of sequential linear system solvers which have been extensively documented in the literature are not applicable to distributed memory systems. This paper presents results and discussions on the selection of simple robust algorithms, easily implemented on specific hardware topologies and provides information for extending the selected methods to different parallel machines. The results can also serve as a reference for parallel computer benchmarking.

Parallel computation of fourier transform on distributed memory computer system

Multicomputer systems (distributed memory computer systems) are becoming more and more popular and will be wildly used in scientific researches. In this paper, we present a parallel algorithm of Fourier Transform of a vector of complex numbers on multicomputer system and give its computing times and its speedup in parallel environment supported by EXPRESS system on the multicomputer system which consists of four SGI workstations. Our analysis shows that the results is ideal and this scheme is suitable to multicomputer systems.

Experience in implementing the Algebraic Multilevel Iteration method on a SIMD-type computer

In this presentation we discuss the Algebraic Multilevel Iteration method and some aspects of its implementation for massively parallel distributed memory computer systems (CM-2, CM-200). Numerical results are presented and compared with those obtained from a serial implementation of the method (IBM RS/6000, SUN SPARC work stations).

Team algorithms in distributed load flow computations

This paper reports the use of a general technique to combine several different methods to solve complex systems of algebraic equations in the context of load flow calculations of electrical power networks. Such a combinations of methods, referred to as 'team algorithms', seem specially well suited to be used with distributed memory computer systems, in an asynchronous environment. Experimental results solving example problems in a commercially available parallel computer system show that a 'synergetic effect' with considerable speedup can be obtained using these 'team algorithms'.

Parallel algorithms for solving large linear systems

The solution of linear systems continues to play an important role in scientific computing. The problems to be solved often are of very large size, so that solving them requires large computer resources. To solve these problems, at least supercomputers with large shared memory or massive parallel computer systems with distributed memory are needed. This paper gives a survey of research on parallel implementation of various direct methods to solve dense linear systems. In particular are considered: Gaussian elimination, Gauss-Jordan elimination and a variant due to Huard (1979), and an algorithm due to Enright (1978), designed in relation to solving (stiff) ODEs, such that stepsize and other method parameters can easily be varied. Some theoretical results are mentioned, including a new result on error analysis of Huard's algorithm. Moreover, practical considerations and results of experiments on supercomputers and on a distributed-memory computer system are presented.