Parallel Programming Paradigms Research Articles

High performance fortran

Fortran 90 provides ideal language support for data parallelism in the form of concurrent array operations. To achieve high efficiency in a massively parallel environment, however, arrays must be optimally distributed over the processing elements. High Performance Fortran is an extension of Fortran 90 that allows such optimum distributions to be specified in cost-effective and portable manner. This paper briefly describes basic parallel architectures and programming paradigms, and sketches the nature of High Performance Fortran.

MP: an application specific concurrent language

In this paper we present the defininition, and implementation of a concurrent language mp (Message Passer), which is aimed at programming embedded multi micro-processor systems. The novel features of the language includes the ability to program and develop the entire distributed system as a single unit, and the provision of simple restricted concurrency constructs that make such systems modular and deadlock free. mp programs are invariant with respect to the characteristics of a particular target machine, for instance, the number of processors, and their mutual connectivity. Programs written in mp can be transformed to a restricted set of Communicating Sequential Processes (CSP), in which particular specifications can be shown to be satisfied. The restricted model of concurrency, does not prevent the language being able to express all of the known course grained parallel programming paradigms. However, the structured nature of the code makes the model particularly attractive in concurrent embedded systems, where deadlock freedom and program correctness are prime issues. An mp program is a collection of objects which execute concurrently, maintain their own local state and connect with other objects via bound variables. Explicit communication between objects is totally transparent to the application programmer. A compiler for the language has been built and a multiprocessor run time environment has been established for any system with a C compiler and the message passing standard MPI library. Several applications have been tested, ranging from a clocked digital circuit simulation, to a simple event driven process controller.

MP: an application specific concurrent language

In the paper the authors present the defininition and implementation of a concurrent language MP (message passer) which is aimed at programming embedded multi-microprocessor systems. The novel features of the language include the ability to program and develop the entire distributed system as a single unit, and the provision of simple restricted concurrency constructs that make such systems modular and deadlock free. MP programs are invariant with respect to the characteristics of a particular target machine, for instance, the number of processors, and their mutual connectivity. Programs written in MP can be transformed to a restricted set of communicating sequential processes (CSP), in which particular specifications can be shown to be satisfied. The restricted model of concurrency does not prevent the language being able to express all of the known course-grained parallel programming paradigms. However, the structured nature of the code makes the model particularly attractive in concurrent embedded systems, where deadlock freedom and program correctness are prime issues. An MP program is a collection of objects which execute concurrently, maintain their own local state and connect with other objects via bound variables. Explicit communication between objects is totally transparent to the application programmer. A compiler for the language has been built and a multiprocessor run-time environment has been established for any system with a C compiler and the message-passing standard MPI library. Several applications have been tested, ranging from a clocked digital circuit simulation, to a simple event-driven process controller.

High performance parallel and distributed computation in compositional CC++

C++ was designed to support a wide range of programming styles. A parallel extension to C++ should support parallelism in all the different programming styles that a C++ program might exploit as well as support a broad range of parallel programming paradigms, such as message passing, active objects and shared memory concurrency. Compositional C++ (CC++) is a parallel C++ that was designed with these goals in mind. In this article, we motivate the design of CC++ and provide a brief overview of the language. We will illustrate how CC++ can be used to define a reusable parallel paradigm libraries and describe how one parallel application used CC++ language features to combine different types of parallelism in a single application.

Generalized speculative computation of parallel simulated annealing

Simulated annealing is known to be highly sequential due to dependences between iterations. While the conventional speculative computation with a binary tree has been found effective for parallel simulated annealing, its performance is limited to (logp)-fold speedup due to parallel execution of logp iterations onp processors. This report presents a new approach to parallel simulated annealing, calledgeneralized speculative computation (GSC). The GSC is synchronous, maintaining the same decision sequence as sequential simulated annealing. The use of two loop indices encoded in a single integer eliminates broadcasting of central data structure to all processors. The master-slave parallel programming paradigm simplifies controlling the activities ofp iterations which are executed in parallel onp processors. To verify the performance of GSC, we implemented 100-city to 500-city Traveling Salesman Problems on the AP1000 massively parallel multiprocessor. Execution results on the AP1000 demonstrate that the GSC approach can indeed be an effective method for parallel simulated annealing as it gave over 20-fold speedup on 100 processors.

Evaluating the performance of parallel programs in a distributed environment

The paper considers methods of evaluating the performance of programs using recent communication harnesses. A demand-based data-farming parallel programming paradigm is used. Possible techniques for performance prediction are examined and a description of a particular method is given, for a shared distributed environment, involving a diffusion approximation. Selected results from a benchmarking study are given to establish the validity of the performance model. A theme of the study is a comparison with a previous transputer-based implementation.

Design of a parallel linear algebra library for verified computation

In this paper we discuss design principles to implement a set of linear algebra subroutines in a portable libraty for parallel computers. Our design supports reuse of code and easy adaption to new parallel programming paradigms or network configurations.

Practical considerations in development of a parallel SKYHI general circulation model

We have developed a parallel version of the SKYHI atmospheric general circulation model. The new parallel model has been designed for shared and distributed memory machines that support data parallel, message passing or worksharing programming paradigms. The newly developed model has a framework that makes the code easier to understand, maintain, and modify, increasing the model's flexibility and scientific productivity. Numerous model changes are described (code design, programming models, language choice, data decomposition, communications, table lookups, memory management, and i/o) that were necessary to develop the model. The performance and verification of the model is described on several systems including a shared-memory machine with high-level worksharing and a distributed-memory system with a data parallel programming paradigm.

Implementing distributed process farms

We report on an experience of implementation of process farms on distributed systems. Rather than focusing on applications, we analyse in detail the techniques we have used for implementing the corresponding support mechanisms. They are actually part of a more general framework that can be easily extended to include other parallel programming paradigms. We try to substantiate the claim that our highly modular structuring may constitute both a practical and powerful approach for several problems of distributed programming support.

Parallel skeletons for structured composition

In this paper, we propose a straightforward solution to the problems of compositional parallel programming by using skeletons as the uniform mechanism for structured composition. In our approach parallel programs are constructed by composing procedures in a conventional base language using a set of high-level, pre-defined, functional, parallel computational forms known as skeletons . The ability to compose skeletons provides us with the essential tools for building further and more complex application-oriented skeletons specifying important aspects of parallel computation. Compared with the process network based composition approach, such as PCN, the skeleton approach abstracts away the fine details of connecting communication ports to the higher level mechanism of making data distributions conform, thus avoiding the complexity of using lower level ports as the means of interaction. Thus, the framework provides a natural integration of the compositional programming approach with the data parallel programming paradigm.

Studies in Computational Science: Parallel Programming Paradigms, Book Review

Studies in Computational Science: Parallel Programming Paradigms, Book Review

PVM-Parallel Virtual Machine: AUsers' Guide and Tutorial for Networked Parallel Computing

Part 1 Introduction: heterogeneous network computing trends in distributed computing PVM overview other packages. Part 2 The PVM system. Part 3 Using PVM: how to obtain the PVM software setup to use PVM setup summary starting PVM common startup problems running PVM programs PVM console details host file options. Part 4 Basic programming techniques: common parallel programming paradigms workload allocation porting existing applications to PVM. Part 5 PVM user interface: process control information dynamic configuration signalling setting and getting options message passing dynamic process groups. Part 6 Program examples: fork-join dot product failure matrix multiply one-dimensional heat equation. Part 7 How PVM works: components messages PVM daemon libpvm library protocols message routing task environment console program resource limitations multiprocessor systems. Part 8 Advanced topics: XPVM porting PVM to new architectures. Part 9 Troubleshooting: geting PVM installed getting PVM running compiling applications running applications debugging and tracing debugging the system. Appendices: history of PVM versions PVM 3 routines.

ASC: an associative-computing paradigm

Today's increased computing speeds allow conventional sequential machines to effectively emulate associative computing techniques. We present a parallel programming paradigm called ASC (ASsociative Computing), designed for a wide range of computing engines. Our paradigm has an efficient associative-based, dynamic memory-allocation mechanism that does not use pointers. It incorporates data parallelism at the base level, so that programmers do not have to specify low-level sequential tasks such as sorting, looping and parallelization. Our paradigm supports all of the standard data-parallel and massively parallel computing algorithms. It combines numerical computation (such as convolution, matrix multiplication, and graphics) with nonnumerical computing (such as compilation, graph algorithms, rule-based systems, and language interpreters). This article focuses on the nonnumerical aspects of ASC. >

Load balancing data parallel programs on distributed memory computers

In this paper a set of programming constructs for the implementation of data parallel algorithms on distributed memory parallel computers is proposed. The load balancing problem for data parallel programs is cast in a special from. Its relation to the general load balancing problem is analyzed. The applicability of these constructs is asserted for a number of grid-oriented numerical applications. A software tool provides run-time support for data parallel programs based on the proposed constructs. While the application - according to the data parallel programming paradigm - partitions the grid, the tool assigns the partitions to the processors, using built-in mapping algorithms. The approach is general enough to accommodate for data parallel algorithms with varying communication structure and variable calculation requirements using pseudo-dynamic load balancing strategies.

Parallelizing algorithms for symbolic computation using MAPLE

||MAPLE|| (speak: parallel Maple) is a portable system for parallel symbolic computation. The system is built as an interface between the parallel declarative programming language Strand and the sequential computer algebra system Maple, thus providing the elegance of Strand and the power of the existing sequential algorithms in Maple. The implementation of different parallel programming paradigms shows that it is fairly easy to parallelize even complex algebraic algorithms using this system. Sample applications (among them algorithms solving multivariate nonlinear equation systems) are implemented on various parallel architectures. For example a straightforward parallelization of the complex and important problem of real root isolation has been parallelized using a generic Strand program of fewer than 20 lines of code and a slight modification of 5 lines in the original sequential Maple source. Even with such a simple modification we gained a speed-up of 5 times, that is better than those reported by others in the literature.

A comparative study of five parallel programming languages

Many different paradigms for parallel programming exist, nearly each of which is employed in dozens of languages. Several researchers have tried to compare these languages and paradigms by examining the expressivity and flexibility of their constructs. Few attempts have been made, however, at practical studies based on actual programming experience with multiple languages. Such a study is the topic of this paper. We will look at five parallel languages, all based on different paradigms. The languages are: SR (based on message passing), Emerald (concurrent objects), Parlog (parallel Horn clause logic), Linda (Tuple Space), and Orca (logically shared data). We have implemented the same parallel programs in each language, using real parallel machines. The paper reports on our experiences in implementing three frequently occurring communication patterns: message passing through a mailbox, one-to-many communication, and access to replicated shared data.

Teaching concurrency with Joyce and Linda

To overcome the difficulty of demonstrating the principles of concurrency with limited parallel computing resources, a concurrent programming language has been designed and implemented which uses the Linda parallel programming paradigm. The language executes under the UNIX operating system and is supported by an X-windows based viewer of programs undergoing concurrent execution. This paper describes the language, its compiler and execution environment and experiences gained in using the language in a third year undergraduate course over the last two years.

Parallel algorithm design for workstation clusters

Clusters of workstations connected by local area networks are in common use in many organizations. The combined processing power of these clusters is rarely exploited owing to the lack of suitable parallel algorithms. The paper describes a parallel programming paradigm called supervisor-worker, suitable for the workstation environment, which can be used to speed up the execution of a large class of existing sequential programs. Simple formulae are developed to predict the speed-up of a parallel algorithm developed in this way. The predictions depend on two easily-determined parameters of the sequential program and the characteristic communication cost of the workstation cluster. Consequently, it is possible to estimate the benefits of the parallel program before proceeding with detailed implementation. As an example, the parallel version of a travelling salesman program is developed and the measured speed-up compared with the predicted speed-up.