Synchronous Parallel Programming Research Articles

Synthesis of Parallel Synchronous Software

In typical embedded applications, the precise execution time of the program does not matter and it is sufficient to meet a real-time deadline. However, modern applications in information security have become much more time-sensitive due to the risk of timing side-channel leakage. The timing of such programs needs to be data-independent and precise. We describe a parallel synchronous software model, which executes as $N$ parallel threads on a processor with word-length $N$ . Each thread is a single-bit synchronous machine with precise, contention-free timing, while each of the $N$ threads still executes as an independent machine. The resulting software supports fine-grained parallel execution. In contrast to earlier work to obtain precise and repeatable timing in software, our solution does not require modifications to the processor architecture nor specialized instruction scheduling techniques. In addition, all threads run in parallel and without contention, which eliminates the problem of thread scheduling. We use hardware (HDL) semantics to describe a thread as a single-bit synchronous machine. Using logic synthesis and code generation, we derive a parallel synchronous implementation of this design. We illustrate the synchronous parallel programming model with practical examples from cryptography and other applications with precise timing requirements.

A scalable method for link prediction in large real world networks

Link prediction has become an important task, especially with the rise of large-scale, complex and dynamic networks. The emerging research area of network dynamics and evolution is directly related to predicting new interactions between objects, a possibility in the near future. Recent studies show that the precision of link prediction can be improved to a great extent by including community information in the prediction methods. As traditional community-based link prediction algorithms can run only on stand-alone computers, they are not well suited for most of the large networks. Graph parallelization can be one solution to such problems. Bulk Synchronous Parallel (BSP) programming model is a recently emerged framework for parallelizing graph algorithms. In this paper, we propose a hybrid similarity measure for link prediction in real world networks. We also propose a scalable method for community structure-based link prediction on large networks. This method uses a parallel label propagation algorithm for community detection and a parallel community information-based Adamic–Adar measure for link prediction. We have developed these algorithms using Bulk Synchronous Parallel programming model and tested them with large networks of various domains.

Mock BSPlib for Testing and Debugging Bulk Synchronous Parallel Software

Testing parallel applications on a large number of processors is often impractical. Not only does it require access to scarce compute resources, but tracking down defects with the available debugging tools can often be very time consuming. Highly parallel codes should be testable on one processor at a time, so that a developer’s workstation is sufficient for executing and debugging test cases on millions of processes. Thanks to its supersteps, Bulk Synchronous Parallel programs are well suited for this kind of testing. This paper presents a mocking library for BSPlib which enables testing of fast and complex parallel algorithms at scale.

MigPF: Towards on self-organizing process rescheduling of Bulk-Synchronous Parallel applications

Migration is a known technique to offer process rescheduling, being especially pertinent for Bulk Synchronous Parallel (BSP) programs. Such programs are organized in a set of supersteps, in which the slowest process always determines the synchronization time. This approach motivated us to develop a first model called MigBSP, which combines computation, communication, and migration costs metrics for process rescheduling decisions. In this paper, a new model named MigPF enhances our previous work in three aspects: (i) a different algorithm for detecting imbalance situations, which considers the performance of each processor, including its enclosed processes, instead of individual times of each process; (ii) an improvement on the rescheduling reactivity through shortening the interval for the next migration call when imbalance situations arise; (iii) a new algorithm for self-organizing the migratable processes and their destinations. Particularly, this third item represents our main scientific contribution, not only in terms of the MigBSP context, but also in a broader one that covers the entire BSP landscape. The algorithm assembles n possible rescheduling plans (where n is the number of processes) and provides a prediction function (pf) that chooses the most profitable rescheduling plan when compared to the current mapping. We developed a MigPF prototype with the Adaptive MPI (AMPI) library, which offers a standard framework for implementing migration-based load balancing policies. We tested this prototype against other built-in AMPI rescheduling policies with two scientific applications: shear sorting and fractal image compression. The results revealed performance gains up to 41% and an overhead limited to 5% when migrations do not take place.

Using Computational Geometry to Improve Process Rescheduling on Round-Based Parallel Applications

Process rescheduling is a known technique to face with system heterogeneity and dynamism, being especially pertinent on Bulk Synchronous Parallel (BSP) programs. These programs are organized in a set of round-based supersteps, in which the slowest process determines the moment of synchronization. This approach motivated us to develop a first model called MigBSP, which combines computation, communication and migration costs metrics for process rescheduling decisions. MigBSP originally employed an heuristic that could select either a single or a collection of process to migrate at each load balancing invocation. The first proposal is not reactive, so you should manually setup a percentage of processes to be migrated as input parameter for the load balancing model. In this work, two novel heuristics, named MigCube and MigHull, are proposed to choose the candidate processes for migration and their destination. Both heuristics consider the use of computational geometry for plotting computation, communication and migration costs metrics in a 3D graph, so both which and where load balancing questions can be answered without any user intervention. We believe that the contribution is not only in the MigBSP landscape, but also for the BSP community, who is trying to enhance performance in round-based applications in an effortless way. In addition to the description of MigCube and MigHull, this article also presents their evaluations with performance gains of up to 42% when enabling process migration over a subset of the Grid5000 infrastructure.

Dynamic thread block launch

GPUs have been proven effective for structured applications that map well to the rigid 1D-3D grid of threads in modern bulk synchronous parallel (BSP) programming languages. However, less success has been encountered in mapping data intensive irregular applications such as graph analytics, relational databases, and machine learning. Recently introduced nested device-side kernel launching functionality in the GPU is a step in the right direction, but still falls short of being able to effectively harness the GPUs performance potential. We propose a new mechanism called Dynamic Thread Block Launch (DTBL) to extend the current bulk synchronous parallel model underlying the current GPU execution model by supporting dynamic spawning of lightweight thread blocks. This mechanism supports the nested launching of thread blocks rather than kernels to execute dynamically occurring parallel work elements. This paper describes the execution model of DTBL, device-runtime support, and microarchitecture extensions to track and execute dynamically spawned thread blocks. Experiments with a set of irregular data intensive CUDA applications executing on a cycle-level simulator show that DTBL achieves average 1.21x speedup over the original flat implementation and average 1.40x over the implementation with device-side kernel launches using CUDA Dynamic Parallelism.

Relation-based computations in a monadic BSP model

We propose a Haskell monadic model of bulk synchronous parallel programs and apply it to the analysis of relation-based computations. Relation-based computations are simple but general patterns found in scientific computing applications. They are easy to implement sequentially, but difficult to parallelize. We use the model to give high-level specifications of distributed relation-based algorithms and outline how to obtain testable parallel implementations from these specifications via equational reasoning. We sketch the architecture of a C++ library of components for distributed relation-based computations. We argue that the model can be used to provide a concise and consistent library documentation.

A Work-Optimal Deterministic Algorithm for the Certified Write-All Problem with a Nontrivial Number of Asynchronous Processors

Martel [C. Martel, A. Park, and R. Subramonian, SIAM J. Comput., 21 (1992), pp. 1070--1099] posed a question for developing a work-optimal deterministic asynchronous algorithm for the fundamental load-balancing and synchronization problem called Certified Write-All (CWA). In this problem, introduced in a slightly different form by Kanellakis and Shvartsman in a PODC'89 paper [P. C. Kanellakis and A. A. Shvartsman, Distributed Computing, 5 (1992), pp. 201--247], p processors must update n memory cells and only then signal the completion of the updates. It is known that solutions to this problem can be used to simulate synchronous parallel programs on asynchronous systems with worst-case guarantees for the overhead of a simulation. Such simulations are interesting because they may increase productivity in parallel computing since synchronous parallel programs are easier to reason about than are asynchronous ones. This paper presents the first solution to the question of Martel, Park, and Subramonian. Specifically, we show a deterministic asynchronous algorithm for the CWA problem. Our algorithm has the work complexity of O(n+p4 log n). This work complexity is asymptotically optimal for a nontrivial number of processors $p \leq \left(n/\log n\right)^{1/4}$. In contrast, all known deterministic algorithms require superlinear in n work when p = n1/r for any fixed $r \geq 1$. Our algorithm generalizes the collision principle introduced by Buss et al. [J. Buss, P. C. Kanellakis, P. L. Ragde, and A. A. Shvartsman, J. Algorithms, 20 (1996), pp. 45--86] in 1996, which has not been previously generalized despite various attempts. Each processor maintains a collection of intervals of {1,2,...,n}. Any processor iteratively selects an interval and works from its tip toward the other tip until it finishes the work or collides with another processor. Collisions are detected effectively using a special Read-Modify-Write operation. In any case, the processor transforms its collection appropriately. Our analysis shows that the transformations preserve some structural properties of collections of intervals. This guarantees that work is assigned to processors in an efficient manner.

A static analysis for Bulk Synchronous Parallel ML to avoid parallel nesting

The BSMLlib library is a library for Bulk Synchronous Parallel (BSP) programming with the functional language Objective Caml. It is based on an extension of the λ -calculus by parallel operations on a data structure named parallel vector, which is given by intention. In order to have an execution that follows the BSP model, and to have a simple cost model, nesting of parallel vectors is not allowed. The novelty of this paper is a type system which prevents such nesting. This system is correct w.r.t. the dynamic semantics.

AN EFFICIENT IMPLEMENTATION OF THE BSP PROGRAMMING LIBRARY FOR VIA

Virtual Interface Architecture(VIA) is a light-weight protocol for protected user-level zero-copy communication. In spite of the promised high performance of VIA, previous MPI implementations for GigaNet's cLAN revealed low communication performance. Two main sources of such low performance are the discrepancy in the communication model between MPI and VIA and the multi-threading overhead. In this paper, we propose a new implementation of the Bulk Synchronous Parallel(BSP) programming library for VIA called xBSP to overcome such problems. To the best of our knowledge, xBSP is the first implementation of the BSP library for VIA. xBSP demonstrates that the selection of a proper library is important to exploit the features of light-weight protocols. Intensive use of Remote Direct Memory Access(RDMA) operations leads to high performance close to the native VIA performance with respect to round trip delay and bandwidth. Considering the effects of multi-threading, memory registration, and completion policy on performance, we could obtain an efficient BSP implementation for cLAN, which was confirmed by experimental results.

Distributed Evaluation of Functional BSP Programs

The BSλp-calculus is a calculus of functional bulk synchronous parallel (BSP) programs. It is the basis for the design of a bulk synchronous parallel ML language. For data-parallel languages, there are two points of view: the programming model where a program is seen as a sequence of operations on parallel vectors, and the execution model where the program is a parallel composition of programs run on each processor of the parallel machine. BSP algorithms are defined by data-parallel algorithms with explicit (physical) processes in order to allow their parallel execution time to be estimated. We present here a distributed evaluation minimally synchronous for BSP execution (which corresponds to the execution model). This distributed evaluation is correct w.r.t. the call-by-value strategy of the BSλp-calculus (which corresponds to the programming model).

DISTRIBUTED EVALUATION OF FUNCTIONAL BSP PROGRAMS

The BS λp-calculus is a calculus of functional bulk synchronous parallel (BSP) programs. It is the basis for the design of a bulk synchronous parallel ML language. For data-parallel languages, there are two points of view: the programming model where a program is seen as a sequence of operations on parallel vectors, and the execution model where the program is a parallel composition of programs run on each processor of the parallel machine. BSP algorithms are defined by data-parallel algorithms with explicit (physical) processes in order to allow their parallel execution time to be estimated. We present here a distributed evaluation minimally synchronous for BSP execution (which corresponds to the execution model). This distributed evaluation is correct w.r.t. the call-by-value strategy of the BS λp-calculus (which corresponds to the programming model).

On the Semantic Foundations of Probabilistic Synchronous Reactive Programs

In this paper we consider synchronous parallel programs ▪ that are composed by sequential randomized processes ▪1,…, ▪k which communicate via shared variables. First, we give an operational semantics for the sequential components ▪i on the basis of a transition relation defined in the classical SOS-style à la Plotkin [Plo81] which we use to specify the behaviour of ▪ by a Markov chain whose transitions stand for the cumulative effect of the activities of the components ▪1,…▪k within one time step. Second, we provide a denotational semantics for ▪ that also models ▪ by a Markov chain. It is based on a (denotational) least fixed point semantics for the sequential components which formalizes the input/output behaviour of the sequential components within one time step. While the operational (declarative) semantics might be the one that a designer (who provides the input for the tool) has in mind, the denotational (procedural) semantics is the one that a compiler might use. We establish a consistency result stating that the Markov chains induced by the operational and denotational semantics are bisimilar in the sense of [LS91].

BSPlib: The BSP programming library

BSPlib is a small communications library for bulk synchronous parallel (BSP) programming which consists of only 20 basic operations. This paper presents the full definition of BSPlib in C, motivates the design of its basic operations, and gives examples of their use. The library enables programming in two distinct styles: direct remote memory access (DRMA) using put or get operations, and bulk synchronous message passing (BSMP). Currently, implementations of BSPlib exist for a variety of modern architectures, including massively parallel computers with distributed memory, shared memory multiprocessors, and networks of workstations. BSPlib has been used in several scientific and industrial applications; this paper briefly describes applications in benchmarking, Fast Fourier Transforms (FFTs), sorting, and molecular dynamics.

BSP linear solvers for dense matrices

This paper presents a study of four parallel linear system solvers implemented using the Bulk Synchronous Parallel programming model. We show that better performances are achieved by the implicit methods due both to higher exploitation of hardware capabilities and to the lower communication costs. A BSP performance model of all the algorithms has been built and the predictions obtained have been compared with the results of measurements on two different parallel systems: a cluster of workstations and a shared memory multiprocessor machine. The implicit algorithms have proved to be from 20% to 40% faster than their traditional counterparts. The high accuracy of prediction makes the BSP programming paradigm an ideal framework for developing parallel algorithms.

An MIMD parallel computer system

The need to run large, compute-bound programs encountered in research, coupled with the high availability of mini- and microcomputers in the laboratory environment, has prompted the linking of independent processors to form multicomputer systems. Important characteristics of the system presented here are the lack of shared memory between processors and the use of purely standard hardware to effect the linking. The resulting MIMD machine is suitable for executing asynchronous and weakly synchronous parallel programs. This is facilatated by assembly language software support to handle communication and to organise the independent sections of executable code for the individual processors. The design principles involved in this hardware configuration and the attendant software are introduced. A brief description of program execution behaviour is given. Applications and examples of programming problems which have been implemented on the system are discussed. An empirical method for assessing the timewise gain which ensues from use of the system is presented and experimental results obtained are outlined.