Distributed Memory Setting Research Articles

A general approach for supporting nonblocking data structures on distributed-memory systems

Nonblocking data structures are an essential part of many parallel applications in that they can help to improve fault tolerance and performance. Although there are scores of nonblocking data structures, such as stacks, queues, double-ended queues (deques), lists, widely used in practice, most of them are designed to be used on shared-memory machines only, and cannot be used in a distributed-memory setting. Several recent studies focus on the development of novel tailor-made nonblocking distributed data structures and omit the potential for adapting a great wealth of existing nonblocking shared-memory ones for the distributed-memory case. Hence, we propose a general approach for bridging the gap between most existing nonblocking data structures and distributed-memory machines in this work. Several challenges, such as safe memory reclamation and solving the ABA problem, must be overcome. To address these issues, we present a global memory management scheme. The scheme takes advantage of hazard pointers which are widely used to tackle the problems in shared-memory environments. To demonstrate our general approach, we take stacks as a typical example of nonblocking data structures. This work also provides a survey of well-known nonblocking stack algorithms along with our analysis and evaluation in distributed-memory environments. Moreover, this paper depicts how to improve performance of a stack algorithm by making use of node locality. Besides, a cost model based on worst cases is devised to help gain a better understanding into experimental results of nonblocking distributed data structures, along with our analysis of the influence of two popular lock-free programming patterns on performance.

Robust and scalable adaptive BDDC preconditioners for virtual element discretizations of elliptic partial differential equations in mixed form

The Virtual Element Method (VEM) is a recent numerical technology for the solution of partial differential equations on computational grids constituted by polygonal or polyhedral elements of very general shape. The aim of this work is to develop effective linear solvers for a general order VEM approximation designed to approximate three-dimensional scalar elliptic equations in mixed form. The proposed Balancing Domain Decomposition by Constraints (BDDC) preconditioner allows to use conjugate gradient iterations, albeit the algebraic linear systems arising from the discretization of the problem are indefinite, ill-conditioned, and of saddle point nature. The condition number of the resulting positive definite preconditioned system is adaptively controlled by means of deluxe scaling operators and suitable local generalized eigenvalue problems for the selection of optimal primal constraints. Numerical results confirm the theoretical estimates and the reliability of the adaptive procedure, with the experimental condition numbers always very close to the prescribed adaptive tolerance parameter. The scalability and quasi-optimality of the preconditioner are demonstrated, and the performances of the proposed solver are compared with state-of-the-art parallel direct solvers and block preconditioning techniques in a distributed memory setting.

An advection–diffusion based filter for machinable designs in topology optimization

This paper introduces a simple formulation for topology optimization problems ensuring manufacturability by machining. The method distinguishes itself from existing methods by using the advection–diffusion equation with Robin boundary conditions to perform a filtering of the design variables. Furthermore, the approach is easy to implement on unstructured meshes and in a distributed memory setting. Finally, the proposed approach can be performed with few to no continuation steps in any system parameters. Applications are demonstrated with topology optimization on unstructured meshes with up to 64 million elements and up to 29 milling tool directions.

Practical Wavelet Tree Construction

We present new sequential and parallel algorithms for wavelet tree construction based on a new bottom-up technique. This technique makes use of the structure of the wavelet trees—refining the characters represented in a node of the tree with increasing depth—in an opposite way, by first computing the leaves (most refined), and then propagating this information upwards to the root of the tree. We first describe new sequential algorithms, both in RAM and external memory. Based on these results, we adapt these algorithms to parallel computers, where we address both shared memory and distributed memory settings. In practice, all our algorithms outperform previous ones in both time and memory efficiency, because we can compute all auxiliary information solely based on the information we obtained from computing the leaves. Most of our algorithms are also adapted to the wavelet matrix , a variant that is particularly suited for large alphabets.

Counting cliques in parallel without a cluster: Engineering a fork/join algorithm for shared-memory platforms

In this paper we develop simple and fast multicore parallel algorithms for counting the number of k-cliques in large undirected graphs, for any small constant k ≥ 4. Clique counting is an important problem in a variety of network analytics applications. Differently from existing solutions, which mainly target distributed memory settings (e.g., MapReduce), our algorithms work on off-the-shelf shared-memory multicore platforms.We assess the effectiveness of our approach through an extensive experimental analysis on a variety of real-world graphs, considering different clique sizes and scalability on different numbers of cores. The experimental results show that our parallel algorithms largely outperform the running times of highly optimized sequential solutions and gracefully scale to non-trivial values of k even on medium/large graphs. For instance, computing hundreds of billions of cliques for rather demanding Web graphs and social networks requires about 15 min on a 32-core machine. As a by-product of our experimental analysis, we also compute the exact number of k-cliques with at most 20 nodes in many real-world networks from the SNAP repository.

Parallelization of continuous and discontinuous Galerkin dual–primal isogeometric tearing and interconnecting methods

In this paper we investigate the parallelization of dual–primal isogeometric tearing and interconnecting (IETI-DP) type methods for solving large-scale continuous and discontinuous Galerkin systems of equations arising from Isogeometric analysis of elliptic boundary value problems. These methods are extensions of the finite element tearing and interconnecting methods to isogeometric analysis. The algorithms are implemented by means of energy minimizing primal subspaces. We discuss how these methods can efficiently be parallelized in a distributed memory setting. Weak and strong scaling studies presented for two and three dimensional problems show an excellent parallel efficiency.

Introduction to the Special Issue on Practical Aspects of High-Level Parallel Programming

From smartphones to supercomputers, parallel architectures are nowadays pervasive. However parallel programming is still reserved to experienced and trained programmers. The trend is towards the increase of cores in processors and the number of processors in multiprocessor machines: The need for scalable computing is everywhere. But parallel and distributed programming is still dominated by low-level techniques such as send/receive message passing and POSIX threads. Thus high-level approaches should play a key role in the shift to scalable computing in every computer and device. Algorithmic skeletons (Google MapReduce being the most well-known skeletal parallelism approach), parallel extensions of functional languages such as Haskell and ML, parallel logic and constraint programming, parallel execution of declarative programs such as SQL queries, meta-programming in object-oriented languages, etc. have produced methods and tools that improve the price/performance ratio of parallel software, and broaden the range of target applications. Also, high level languages offer a high degree of abstraction which ease the development of complex systems. Moreover, being based on formal semantics, it is possible to certify the correctness of critical parts of the applications. The aim of all these languages and tools is to improve and ease the development of applications. The paper presented in this special issue are extended versions of papers presented at the Practical Aspects of High-Level Parallel Programming (PAPP) workshop that was affiliated to the International Conference on Computational Science (ICCS) from 2004 to 2012, and the Practical Aspects of High-Level Parallel Programming (PAPP) track of the ACM Symposium on Applied Computing (SAC). It also features a paper written specifically for this special issue. While a lot of work in high-level approaches focus on regular data structures such as arrays and multi-dimensional arrays, the papers of this issue consider more irregular data structures, in particular trees and graphs. Kaheki, Matsuzaki and Emoto present a new approach for parallel reduction on trees in a distributed memory setting. Their algorithm is based on a suitable serialization of trees and its efficiency is shown for the Bulk Synchronous Parallel (BSP) model. Matzusaki proposes a set of efficient skeletons on distributed trees: their efficient implementation relies on the segmentation of trees based on the idea of m-bridges that ensures high locality, and allows to represent local segments as serialized arrays for high sequential performance. Dazzi, Carlini, Lulli and Ricci propose Telos, a high-level approach for large graphs processing based on overlay networks, and an implementation on top of Apache Spark. Hajibaba, Gorgin, and Sharifi present a sequence similarity parallelization technique in the context of another Apache project: Storm a stream processing framework. I wish to thank Anne Benoit and Frederic Gava, co-chairs of some the previous PAPP workshops and the program committee members of the PAPP workshops and the PAPP ACM SAC track who reviewed the initial papers and the revised versions of this special issue.

Parallel multiphase field simulations with OpenPhase

The open-source software project OpenPhase allows the three-dimensional simulation of microstructural evolution using the multiphase field method. The core modules of OpenPhase and their implementation as well as their parallelization for a distributed-memory setting are presented. Especially communication and load-balancing strategies are discussed. Synchronization points are avoided by an increased halo-size, i.e. additional layers of ghost cells, which allow multiple stencil operations without data exchange. Load-balancing is considered via graph-partitioning and sub-domain decomposition. Results are presented for performance benchmarks as well as for a variety of applications, e.g. grain growth in polycrystalline materials, including a large number of phase fields as well as Mg–Al alloy solidification. Program summaryProgram Title: OpenPhaseProgram Files doi:http://dx.doi.org/10.17632/2mnv2fvkkk.1Licensing provisions: GPLv3Programming language: C++Nature of problem: OpenPhase[1] allows the simulation of microstructure evolution during materials processing using the multiphase field method. In order to allow an arbitrary number of phase fields active parameter tracking is used, which can cause load imbalances in parallel computations.Solution method: OpenPhase solves the phase field equations using an explicit finite difference scheme. The parallel version of OpenPhase provides load-balancing using over-decomposition of the computational domain and graph-partitioning. Adaptive sub-domain sizes are used to minimize the computational overhead of the over-decomposition, while allowing appropriate load-balance.Additional comments including Restrictions and Unusual features: The distributed-memory parallelism in OpenPhase uses MPI. Shared-memory parallelism is implemented using OpenMP. The library uses C++11 features and therefore requires GCC version 4.7 or higher. [1] www.openphase.de

Distributed-memory parallelization of the Wigner Monte Carlo method using spatial domain decomposition

The Wigner Monte Carlo method, based on the generation and annihilation of particles, has emerged as a promising approach to treat transient problems of quantum electron transport in nanostructures. Tackling these simulations in multiple spatial dimensions demands a parallelized approach to facilitate a practical application of the method in order to investigate realistic problems, due to the otherwise exorbitant execution-times and memory requirements. Because of the annihilation step, a straight-forward parallelization of the Wigner Monte Carlo code is not possible, since sub-ensembles of particles can not be treated independently. Moreover, the large memory requirements of the annihilation procedure presents challenges when working in a distributed-memory setting. A solution to this problem is presented here with a parallelization approach using a spatial domain decomposition, implemented using the message passing interface. The presented benchmark results, based on standard one-dimensional examples, exhibit a good efficiency in the scalability of not only speed, but also memory consumption, which is paramount for the simulation of realistic devices.

NOMAD

We develop an efficient parallel distributed algorithm for matrix completion, named NOMAD (Non-locking, stOchastic Multi-machine algorithm for Asynchronous and Decentralized matrix completion). NOMAD is a decentralized algorithm with non-blocking communication between processors. One of the key features of NOMAD is that the ownership of a variable is asynchronously transferred between processors in a decentralized fashion. As a consequence it is a lock-free parallel algorithm. In spite of being asynchronous, the variable updates of NOMAD are serializable, that is, there is an equivalent update ordering in a serial implementation. NOMAD outperforms synchronous algorithms which require explicit bulk synchronization after every iteration: our extensive empirical evaluation shows that not only does our algorithm perform well in distributed setting on commodity hardware, but also outperforms state-of-the-art algorithms on a HPC cluster both in multi-core and distributed memory settings.

A distributed algorithm for strong bisimulation reduction of state spaces

It is a known problem that state spaces can grow very large, which makes operating on them (including reducing them) difficult because of operational memory shortage. In an attempt to extend the size of the state spaces that can be dealt with, we designed and implemented a bisimulation reduction algorithm for distributed memory settings using message passing communication. By using message passing, the same implementation can be used on both clusters of workstations and large shared memory machines. The algorithm performs reduction of large labeled transition systems modulo strong bisimulation. We justify its correctness and termination and provide an evaluation of the worst-case time and message complexity and some performance data from a prototype implementation. Both theory and practice show that the algorithm scales up with the number of workstations.

A Distributed Algorithm for Strong Bisimulation Reduction of State Spaces

It is a known problem that state spaces can grow very big, which makes operating with them (including reducing them) difficult because of memory shortage. In the attempt to extend the size of the state spaces that can be dealt with, we designed and implemented a bisimulation reduction algorithm for distributed memory settings using message passing communication. By using message passing, the same implementation can be used on both large SMP machines and clusters of workstations. The algorithm performs reduction of large labeled transition systems modulo strong bisimulation. We justify its correctness and termination. We provide an evaluation of the worst-case time and message complexity and some performance data from a prototype implementation. Both theory and practice show that the algorithm scales up with the number of workers.

D-Dimensional Range Search on Multicomputers

The range tree is a fundamental data structure for multidimensional point sets, and, as such, is central in a wide range of geometric and database applications. In this paper we describe the first nontrivial adaptation of range trees to the parallel distributed memory setting (BSP-like models). Given a set of n points in d -dimensional Cartesian space, we show how to construct on a coarse-grained multicomputer a distributed range tree T in time O( s / p + T c (s,p)) , where s = n log d-1 n is the size of the sequential data structure and T c (s,p) is the time to perform an h -relation with h=Θ (s/p) . We then show how T can be used to answer a given set Q of m=O(n) range queries in time O((s log m)/p + T c (s,p)) and O((s log m)/p + T c (s,p) + k/p) , where k is the number of results to be reported. These parallel construction and search algorithms are both highly efficient, in that their running times are the sequential time divided by the number of processors, plus a constant number of parallel communication rounds.

Approximate sparsity patterns for the inverse of a matrix and preconditioning

We consider a general sparse matrix A. Computing a sparse approximate inverse matrix M by minimizing ∥ AM− E∥ in the Frobenius norm is very useful for deriving preconditioners in iterative solvers, especially in a parallel environment. The problems, that appear in this connection in a distributed memory setting, are the distribution of the data—mainly submatrices of A—on the different processors. An a-priori knowledge of the data that has to be sent to a processor would be very helpful in this connection in order to reduce the communication time. In this paper, we compare different strategies for choosing a-priori an approximate sparsity structure of A −1. Such a sparsity pattern can be used as a maximum pattern of allowed entries for the sparse approximate inverse M. Then, with this maximum pattern we are able to distribute the claimed data in one step to each processor. Therefore, communication is necessary only at the beginning and at the end of a subprocess. Using the characteristic polynomials and the Neumann series associated with A and A T A, we develop heuristic methods to find good and sparse approximate patterns for A −1. Furthermore, we exactly determine the submatrices that are used in the SPAI algorithm to compute one new column of the sparse approximate inverse. Hence, it is possible to predict in advance an upper bound for the data that each processor will need. Based on numerical examples we compare the different methods with regard to the quality of the resulting approximation M.

Mobile-Process-Based Parallel Simulation

Our focus is on the novel use of a process-oriented methodology in software systems for parallel simulation on distributed memory. To the best of our knowledge, the few existing systems which adopt a process view strictly use message passing to effect process interaction in distributed-memory settings. As a result, these systems avoid scenarios in which processes directly access passive but shared components. This can restrict the manner in which a system is modeled and hinder the phase of distributed model construction. In this paper, we propose an approach which utilizes mobile processes in distributed-memory simulation systems. The approach entails the migration of a requesting process with its timestamp to the remote site hosting the requested passive object. Major advantages of this approach include one-time transmission, fixed communication topology, and increased locality of reference. Empirical results based on lightweight processes show that the mobile process paradigm can be as efficient as the message-passing paradigm.

Parallel implicit unstructured grid Euler solvers

A mesh-vertex finite volume scheme for solving the Euler equations on triangular unstructured meshes is implemented on a multiple-instruction/multiple-data stream parallel computer. An explicit four-stage Runge-Kutta scheme is used to solve two-dimensional flow problems. A family of implicit schemes is also developed to solve these problems, where the linear system that arises at each time step is solved by a preconditioned GMRES algorithm. Two partitioning strategies are employed: one that partitions triangles and the other that partitions vertices. The choice of the preconditioner in a distributed memory setting is discussed. All of the methods are compared both in terms of elapsed times and convergence rates. It is shown that the implicit schemes offer adequate parallelism at the expense of minimal sequential overhead. The use of a global coarse grid to further minimize this overhead is also investigated. The schemes are implemented on a distributed memory parallel computer, the Intel iPSC/860.