Bulk Synchronous Parallel Programming Research Articles

Automatic Cost Analysis for Imperative BSP Programs

Bulk Synchronous Parallel (BSP) is a model for parallel computing with predictable scalability. BSP has a cost model: programs can be assigned a cost which describes their resource usage on any parallel machine. However, the programmer has to manually derive this cost. This paper describes an automatic method for the derivation of BSP program costs, based on classic cost analysis and approximation of polyhedral integer volumes. Our method requires and analyzes programs with textually aligned synchronization and textually aligned, polyhedral communication. We have implemented the analysis and our prototype obtains cost formulas that are parametric in the input parameters of the program and the parameters of the BSP computer and thus bound the cost of running the program with any input on any number of cores. We evaluate the cost formulas and find that they are indeed upper bounds, and tight for data-oblivious programs. Additionally, we evaluate their capacity to predict concrete run times in two parallel settings: a multi-core computer and a cluster. We find that when exact upper bounds can be found, they accurately predict run-times. In networks with full bisection bandwidth, as the BSP model supposes, results are promising with errors < 50%.

A scalable method for link prediction in large real world networks

Link prediction has become an important task, especially with the rise of large-scale, complex and dynamic networks. The emerging research area of network dynamics and evolution is directly related to predicting new interactions between objects, a possibility in the near future. Recent studies show that the precision of link prediction can be improved to a great extent by including community information in the prediction methods. As traditional community-based link prediction algorithms can run only on stand-alone computers, they are not well suited for most of the large networks. Graph parallelization can be one solution to such problems. Bulk Synchronous Parallel (BSP) programming model is a recently emerged framework for parallelizing graph algorithms. In this paper, we propose a hybrid similarity measure for link prediction in real world networks. We also propose a scalable method for community structure-based link prediction on large networks. This method uses a parallel label propagation algorithm for community detection and a parallel community information-based Adamic–Adar measure for link prediction. We have developed these algorithms using Bulk Synchronous Parallel programming model and tested them with large networks of various domains.

Mock BSPlib for Testing and Debugging Bulk Synchronous Parallel Software

Testing parallel applications on a large number of processors is often impractical. Not only does it require access to scarce compute resources, but tracking down defects with the available debugging tools can often be very time consuming. Highly parallel codes should be testable on one processor at a time, so that a developer’s workstation is sufficient for executing and debugging test cases on millions of processes. Thanks to its supersteps, Bulk Synchronous Parallel programs are well suited for this kind of testing. This paper presents a mocking library for BSPlib which enables testing of fast and complex parallel algorithms at scale.

Automated Generation of BSP Automata

Bulk-Synchronous Parallel (BSP) is a bridging model between abstract execution and concrete parallel architectures that retains enough information to model performance scalability. In order to model BSP program executions, Hains adapted the theory of finite automata to the BSP paradigm by introducing BSP automata [12]. In the initial description of the theory, BSP automata had to be explicitly defined as structured sets of states and transitions. The lack of a clean and efficient algorithm for generating them from regular expressions would have prevented this theory from being used in practice. To alleviate this problem we introduce in this paper an algorithm that generates a BSP automaton recognizing a BSP language defined by a BSP regular expression. The main practical use of BSP automata developed in this paper is the parallelization of finite state automata with an explicit distribution and a performance model, that enable parallel matching of regular expressions. Secondarily, BSP regular expressions provide a convenient structure for automatic code generation of imperative BSP programs that is also developed in this paper.

Compiler Optimization for Reducing Leakage Power in Multithread BSP Programs

Multithread programming is widely adopted in novel embedded system applications due to its high performance and flexibility. This article addresses compiler optimization for reducing the power consumption of multithread programs. A traditional compiler employs energy management techniques that analyze component usage in control-flow graphs with a focus on single-thread programs. In this environment the leakage power can be controlled by inserting on and off instructions based on component usage information generated by flow equations. However, these methods cannot be directly extended to a multithread environment due to concurrent execution issues. This article presents a multithread power-gating framework composed of multithread power-gating analysis (MTPGA) and predicated power-gating (PPG) energy management mechanisms for reducing the leakage power when executing multithread programs on simultaneous multithreading (SMT) machines. Our multithread programming model is based on hierarchical bulk-synchronous parallel (BSP) models. Based on a multithread component analysis with dataflow equations, our MTPGA framework estimates the energy usage of multithread programs and inserts PPG operations as power controls for energy management. We performed experiments by incorporating our power optimization framework into SUIF compiler tools and by simulating the energy consumption with a post-estimated SMT simulator based on Wattch toolkits. The experimental results show that the total energy consumption of a system with PPG support and our power optimization method is reduced by an average of 10.09% for BSP programs relative to a system without a power-gating mechanism on leakage contribution set to 30%; and the total energy consumption is reduced by an average of 4.27% on leakage contribution set to 10%. The results demonstrate our mechanisms are effective in reducing the leakage energy of BSP multithread programs.

REDUCING THE BULK IN THE BULK SYNCHRONOUS PARALLEL MODEL

For over two decades the dominant means for enabling portable performance of computational science and engineering applications on parallel processing architectures has been the bulk-synchronous parallel programming (BSP) model. Code developers, motivated by performance considerations to minimize the number of messages transmitted, have typically pursued a strategy of aggregating message data into fewer, larger messages. Emerging and future high-performance architectures, especially those seen as targeting Exascale capabilities, provide motivation and capabilities for revisiting this approach. In this paper we explore alternative configurations within the context of a large-scale complex multi-physics application and a proxy that represents its behavior, presenting results that demonstrate some important advantages as the number of processors increases in scale.

To 4,000 compute nodes and beyond

The explosive growth of "big data" is giving rise to a new breed of large scale graph systems, such as Pregel. This poster describes our ongoing work in characterizing and minimizing the communication cost of Bulk Synchronous Parallel (BSP) graph mining systems, like Pregel, when scaling to 4,096 compute nodes. Existing implementations generally assume a fixed communication cost. This is sufficient in small deployments as the BSP programming model (i.e., overlapping computation and communication) masks small variations in the underlying network. In large scale deployments, such variations can dominate the overall runtime characteristics. In this poster, we first quantify the impact of network communication on the total compute time of a Pregel system. We then propose an efficient vertex placement strategy that subsamples highly connected vertices and applies the Reverse Cuthill-McKee (RCM) algorithm to efficiently partition the input graph and place partitions closer to each other based on their expected communication patterns. We finally describe a vertex replication strategy to further reduce communication overhead.

CPS implementation of a BSP composition primitive with application to the implementation of algorithmic skeletons

Bulk-synchronous parallel ML (BSML) is an ML-based language designed to code bulk synchronous parallel (BSP) algorithms. It allows an estimation of execution time, and avoids deadlocks and non-determinism. BSML proposes an extension of ML programing with a small set of primitives. One of these primitives, called parallel superposition, allows the parallel composition of two BSP programs. Nevertheless, its past implementation uses system threads and has a serious drawback, which is the cost of managing threads in ML-like languages. This work presents a new implementation of this primitive based on a continuation-passing-style transformation guided by a flow analysis. To test it and show its usefulness, we also have implemented the OCamlP3L's algorithmic skeletons and compared their efficiencies with the original ones. The work presented here is tightly related to the BSP model, but is not specific to ML. Hence, we reckon there would be little work involved in translating it to, for instance, Java or Python.

Relation-based computations in a monadic BSP model

We propose a Haskell monadic model of bulk synchronous parallel programs and apply it to the analysis of relation-based computations. Relation-based computations are simple but general patterns found in scientific computing applications. They are easy to implement sequentially, but difficult to parallelize. We use the model to give high-level specifications of distributed relation-based algorithms and outline how to obtain testable parallel implementations from these specifications via equational reasoning. We sketch the architecture of a C++ library of components for distributed relation-based computations. We argue that the model can be used to provide a concise and consistent library documentation.

Barrier synchronisation: Axiomatisation and relaxation

Abstract. The bulk synchronous parallel (BSP) model is underpinned by a global synchronisation operation. This operation both synchronises processes and updates local stores using asynchronous communications. In this paper a generalised, nondeterministic, substitution axiom (over functions) is identified with BSP barrier synchronisation. Correctness proofs of BSP programs based on the axiom are structurally similar to proofs of correctness of sequential programs. Global synchronisation is often considered to be relatively expensive to implement on distributed architectures for applications having low levels of communications. In an extreme situation a process having no external interactions may have to wait unnecessarily at a barrier. In this paper a form of barrier operation is introduced which is straightforward to reason about and, with a view to increasing the potential for efficient implementation, permits barrier operations to be relaxed through partial uncoupling thereby allowing processes which do not need to synchronise to proceed.

Distributed Evaluation of Functional BSP Programs

The BSλp-calculus is a calculus of functional bulk synchronous parallel (BSP) programs. It is the basis for the design of a bulk synchronous parallel ML language. For data-parallel languages, there are two points of view: the programming model where a program is seen as a sequence of operations on parallel vectors, and the execution model where the program is a parallel composition of programs run on each processor of the parallel machine. BSP algorithms are defined by data-parallel algorithms with explicit (physical) processes in order to allow their parallel execution time to be estimated. We present here a distributed evaluation minimally synchronous for BSP execution (which corresponds to the execution model). This distributed evaluation is correct w.r.t. the call-by-value strategy of the BSλp-calculus (which corresponds to the programming model).

A NEW WAY TO DIVIDE AND CONQUER

Valiant's model of bulk-synchronous parallel (BSP) computation does not allow the programmer to synchronize a subset, rather than the complete set of a parallel computer's processors. This is perceived by many to be an obstacle to expressing divide-and-conquer algorithms in the BSP model. We argue that the divide-and-conquer paradigm fits naturally into the BSP model, without any need for subset synchronization. The proposed method of divide-and-conquer BSP programming is fully compliant with the BSP computation model. The method is based on sequentially interleaved threads of BSP computation, called superthreads.

New Way to Divide and Conquer

Valiant's model of bulk-synchronous parallel (BSP) computation does not allow the programmer to synchronize a subset, rather than the complete set of a parallel computer's processors. This is perceived by many to be an obstacle to expressing divide-and-conquer algorithms in the BSP model. We argue that the divide-and-conquer paradigm fits naturally into the BSP model, without any need for subset synchronization. The proposed method of divide-and-conquer BSP programming is fully compliant with the BSP computation model. The method is based on sequentially interleaved threads of BSP computation, called superthreads.

JBSP: A BSP Programming Library in Java

In this paper, we introduce a Java implementation of the Bulk Synchronous Parallel (BSP) model. JBSP (a Java-based BSP system) uses a two-daemon architecture which makes a clear separation of the computation and communication involved in parallel programs. Java threads are used in the implementation of the JBSP system to realize user defined JBSP tasks as well as to carry out system activities. The JBSP library provides programmers with both explicit message-passing and remote memory access communication methods. In addition, a set of benchmark experiments was also carried out to compare the performance of JBSP with that of BSPlib. Results show that there is considerable overhead in communications in the current implementation of JBSP due to the use of Java's Object Serialization. It was also found that although BSPlib outperforms JBSP in all the cases, JBSP is more scalable in terms of the BSP machine-parameters.

AXIOMATIC FRAMEWORKS FOR DEVELOPING BSP-STYLE PROGRAMS*

In bulk-synchronous parallel (BSP) computation a superstep comprises a collection of concurrently executed processes with initial and terminal synchronisations. Data transfer between processes is realised through asynchronous communications. BSP programs can be organised either as explicit compositions of supersteps or as parallel compositions of threads (processes) which include synchronisation alignment operations. In this paper axiomatic semantics for the two approaches are proposed: in both cases the semantics are based on a new form of multiple substitution - predicate substitution which generalises previous definitions of substitution. Predicate substitution together with global synchronisation provide a means of linking state based and process semantics of BSP. Features of both models are illustrated through correctness proofs of matrix multiplication programs

PRAM programming: in theory and in practice

That the influence of the PRAM model is ubiquitous in parallel algorithm design is as clear as the fact that it is technologically infeasible for the forseeable future. The current generation of parallel hardware prominently features distributed memory and high-performance interconnection networks—very much the antithesis of the shared memory required for the PRAM model. It has been shown that, in spite of communication costs, for some problems very fast parallel algorithms are available for distributed-memory machines—from embarassingly parallel problems to sorting and numerical analysis. In contrast it is known that for other classes of problem PRAM-style shared-memory simulation on a distributed-memory machine can, in theory, produce solutions of comparable performance to the best possible for such architectures. The Bulk Synchronous Parallel (BSP) model accurately represents most parallel machines—theoretical and actual—in an execution and cost model. We introduce a scalable portable PRAM realization appropriate for BSP computers and a methodology for usage. Our system is fast and built upon the familiar sequential C++ coupled with the new standard BSP library of parallel computation and communication primitives. It is portable to and predictable on a vast number of parallel computers including workstation clusters, a 256-processor Cray T3D, an 8-node IBM SP/2 and a 4-node shared-memory SGI Power Challenge machine. Our approach achieves simplicity of programming over direct-mode BSP programming for reasonable overhead cost. We objectively compare optimized BSP and PRAM algorithms implemented with our C++ PRAM library and provide encouraging experimental results for our new style of programming. Copyright © 2000 John Wiley & Sons, Ltd.

BSPlib: The BSP programming library

BSPlib is a small communications library for bulk synchronous parallel (BSP) programming which consists of only 20 basic operations. This paper presents the full definition of BSPlib in C, motivates the design of its basic operations, and gives examples of their use. The library enables programming in two distinct styles: direct remote memory access (DRMA) using put or get operations, and bulk synchronous message passing (BSMP). Currently, implementations of BSPlib exist for a variety of modern architectures, including massively parallel computers with distributed memory, shared memory multiprocessors, and networks of workstations. BSPlib has been used in several scientific and industrial applications; this paper briefly describes applications in benchmarking, Fast Fourier Transforms (FFTs), sorting, and molecular dynamics.

BSP linear solvers for dense matrices

This paper presents a study of four parallel linear system solvers implemented using the Bulk Synchronous Parallel programming model. We show that better performances are achieved by the implicit methods due both to higher exploitation of hardware capabilities and to the lower communication costs. A BSP performance model of all the algorithms has been built and the predictions obtained have been compared with the results of measurements on two different parallel systems: a cluster of workstations and a shared memory multiprocessor machine. The implicit algorithms have proved to be from 20% to 40% faster than their traditional counterparts. The high accuracy of prediction makes the BSP programming paradigm an ideal framework for developing parallel algorithms.

The bulk-synchronous parallel random access machine

The model of bulk-synchronous parallel (BSP) computation is an emerging paradigm of general-purpose parallel computing. Originally, BSP was defined as a distributed memory model. Shared-memory style BSP programming had to be provided by PRAM simulation. However, this approach destroys data locality and therefore may prove inefficient for many practical problems. In this paper we present a new BSP-type model, called BSPRAM, which reconciles sharedmemory style programming with efficient exploitation of data locality. BSPRAM can be optimally simulated by BSP for a broad range of algorithms. We identify some characteristic properties of such algorithms: obliviousness, slackness, granularity. Finally, we illustrate these concepts by presenting BSPRAM algorithms for butterfly dag computation, cube dag computation, dense matrix multiplication and sorting.

Scalability, portability and predictability: The BSP approach to parallel programming

General purpose parallel computing systems come in a variety of forms. We have various kinds of distributed memory architectures, shared memory multiprocessors, and clusters of workstations. New technologies may increase this range still further. Can one hope to design portable and scalable parallel software in the face of such architectural diversity? In this paper we show that it is indeed possible to produce fully portable parallel software which will run with highly efficient, scalable and predictable performance on any general purpose parallel architecture. The approach we describe is based on the bulk synchronous parallel (BSP) model of computation. The BSP model provides a simple, unified framework for the design and programming of all kinds of general purpose parallel systems. Over the last few years, a number of important research activities in algorithms and architectures have been pursued as part of this new approach to scalable parallel computing. In this paper we give some simple BSP algorithms and show how they can be expressed as programs. We also briefly describe some of the BSP programming language developments which are now being pursued.