Execution Of Parallel Programs Research Articles

Prophet: automated scheduling of SPMD programs in workstation networks

Obtaining efficient execution of parallel programs in workstation networks is a difficult problem for the user. Unlike dedicated parallel computer resources, network resources are shared, heterogeneous, vary in availability, and offer communication performance that is still an order of magnitude slower than parallel computer interconnection networks. Prophet, a system that automatically schedules data parallel SPMD programs in workstation networks for the user, has been developed. Prophet uses application and resource information to select the appropriate type and number of workstations, divide the application into component tasks and data across these workstations, and assign tasks to workstations. This system has been integrated into the Mentat parallel processing system developed at the University of Virginia. A suite of scientific Mentat applications has been scheduled using Prophet on a heterogeneous workstation network. The results are promising and demonstrate that scheduling SPMD applications can be automated with good performance. Copyright © 1999 John Wiley & Sons, Ltd.

Replication Techniques in Distributed Systems

i»? A. A. Helal, A. A. Heddaya, and B. B. Bhargrava Kluwer Academic Publishers, Boston, 1996, 176 pp. ISBN 0-7923-9800-9, $118.50 Computing in the 1990s has reached the state of distributed computing. A basis of this form of computing is a distributed computing system, which is built on the following three components: (a) personal computers, (b) local and fast wide area networks, and (c) system and application software. By amalgamating computers and networks into one single computing system and providing an appropriate system software, a distributed computing system has created the possibility of sharing information and peripheral resources. Furthermore, these systems improved performance of a computing system and individual users through parallel execution of programs, load balancing and sharing, and replication of programs and data. Distributed computing systems are also characterized by enhanced availability, and increased reliability. Replication is the key to providing high availability, fault tolerance, and enhanced performance in a distributed computing system. As companies move toward systems that are more open and distributed, replication is becoming increasingly important in the ability to provide data and services that are current, correct and available, which is a key factor in maintaining a competitive advantage over rivals. However, replication has also generated some serious challenges. For example, when a replica is updated, how do we propagate such an update to other replicas? If multiple replicas are updated simultaneously and some of the updates conflict one another, how do we resolve the conflict? How do we deal with the situation where a replica is down and then subsequently recovered? How do we deal with network partitions? Not surprisingly, considerable research efforts have been directed towards the solution of these challenges. However, most of the research results are scattered among many journals, conference proceedings, theses, and technical reports. The book by Helal, Heddaya and Bhargava has gathered the best material of these research results and formed a coherent collection that includes definitions, theoretical background, algorithms, annotated bibliographies of commercial and experimental prototype systems, and more than 200 references. This book classifies the entities that can be replicated on a distributed computing system into the following categories: (a) data, (b) process, (c) objects, and (d) messages. After the introduction on the goals and main approaches of replication techniques, the book uses individual chapters to describe major methods for each of the above replication categories. The main focus of this part, however, is on the replication of data. The book also contains two chapters on the replication issues in heterogeneous, mobile, and large-scale systems, and on the future of replication techniques. Apart from the normal chapters, the book contains a rich set of appendices that are very useful for further studies of replication techniques. These appendices include brief descriptions of two dozen commercial and experimental systems that use replication techniques, annotated bibliographies of selected literature on various topics of replication, and an introduction on serializability theory. Each annotated bibliography contains an overview of the specific topic and is written by an invited expert in that field. The combination of the normal chapters and appendices cover the entire spectrum of replication, from definitions, concepts, theories, algorithms, techniques, to systems. The book serves as an excellent introduction and roadmap to the issues of replication. Although it does not provide a detailed description of every replication techniques and systems, the book covers most of the fundamental work that allows the reader to understand the roots of these techniques and systems, and to find the ways to search for the details. This book is a valuable reference book for anyone studying replication techniques or implementing replication systems, specifically, it can be very useful to post-graduate students, practitioners, and beginning researchers of this exciting field. Wanlei Zhou, School of Computing and Mathematics, Deakin University

A Heterogeneous Mixed-Mode Execution Model for Massively Parallel Systems

In this paper, we consider a massively parallel system that is composed of heterogeneous processors, that is, processors with different processing power, and that combines the advantages of the SIMD and MIMD architectures. The heterogeneous mixed-mode (HeMM) execution model is composed of two main components, which operate in the well-known SIMD and MIMD paradigms. The main computing power comes from a component that is composed of a massive number of processors and operates in a data parallel manner. The other component is composed of a few (or even one) fast processors which operate in the MIMD paradigm. The operation of a small number of processors in an MIMD paradigm has been well demonstrated through actual systems. The processors in this component add flexibility to the execution of the parallel programs such that it adjusts to the changing parallelism of the program to enhance the performance. Based on this execution model we analyze the gains in performance that is obtainable by this new system. We show that substantial performance gains can be obtained by using the HeMM system.

Space efficient execution of deterministic parallel programs

We model a deterministic parallel program by a directed acyclic graph of tasks, where a task can execute as soon as all tasks preceding it have been executed. Each task can allocate or release an arbitrary amount of memory (i.e., heap memory allocation can be modeled). We call a parallel schedule efficient if the amount of memory required is at most equal to the number of processors times the amount of memory required for some depth-first execution of the program by a single processor. We describe a simple, locally depth-first scheduling algorithm and show that it is always space efficient. Since the scheduling algorithm is greedy, it will be within a factor of two of being optimal with respect to time. For the special case of a program having a series-parallel structure, we show how to efficiently compute the worst case memory requirements over all possible depth-first executions of a program. Finally, we show how scheduling can be decentralized, making the approach scalable to a large number of processors when there is sufficient parallelism.

Exploiting value locality to exceed the dataflow limit

The serialization constraints imposed by true data dependences have always been regarded as an absolute dataflow limit on the parallel execution of serial programs. This paper describes value prediction, a new technique that allows data dependent instructions to issue and execute in parallel without violating program semantics. This technique exploits value locality, or the likelihood of the recurrence of a previously-seen value within a storage location inside a computer system. Value prediction consists of predicting entire 32- and 64-bit register values based on previously-seen values. We find that values loaded from memory or generated by ALU instructions are frequently predictable. Furthermore, we show that simple microarchitectural enhancements to a modern microprocessor implementation based on the PowerPC 620 that enable value prediction can effectively exploit value locality to collapse true dependences, reduce average memory and result latencies, and provide average performance gains of 3%-23% by exceeding the dataflow limit.

Automatic Generation of Modular Time-Space Mappings and Data Alignments

Time-space transformations and data alignments that can lead to efficient execution of parallel programs have been extensively studied. Recently, modular time-space transformations have been proposed to generate a class of algorithm mappings that cannot be described by linear time-space transformations. This paper proposes a new class of data alignments, called expanded modular data alignments (EMDAs), for programs that result from modular time-space transformations. An EMDA subsumes multiple modular data alignments, which are described by affine functions modulo a constant vector. Conditions of a modular time-space mapping and an EMDA for perfect alignment are described. However, these conditions together with other conditions for validity and optimality of a modular mapping introduce nonlinear constraints in the problem of generating modular mappings. A method of O(n^2) complexity is provided to choose some entries of a transformation matrix so that nonlinear constraints are transformed into linear ones, where n is the dimension of the computation domain (e.g., the number of nested loops). Although the solution space of the problem is reduced by assigning fixed values to some entries, the proposed heuristic attempts to minimize the number of the fixed entries and consequently to exclude as few solutions as possible.

Efficient Execution of Nondeterministic Parallel Programs on Asynchronous Systems

We consider the problem of asynchronous execution of parallel programs. We assume that the original program is designed for a synchronous system, whereas the actual system may be asynchronous. We seek an automatic execution scheme , which allows the asynchronous system to execute the synchronous program. Previous execution schemes provide solutions only for the case where the original program is deterministic. Here, we provide the first solution for the more general case where the original program can be nondeterministic (e.g., randomized). Our scheme is based on a novel agreement protocol for the asynchronous parallel setting. Our protocol allows n asynchronous processors to agree on n word-sized values in O ( n log n log log n ) total work , assuming an oblivious adversary scheduler. Total work is defined to be the summation of the number of steps performed by all processors (including steps from busy waiting).

Using Java and JavaScript in the Virtual Programming Laboratory: a Web-based parallel programming environment

The Virtual Programming Laboratory (VPL) is a Web-based virtual programming environment built based on a client–server architecture. The system can be accessed on any platform (Unix, PC or Mac) using a standard Java-enabled browser. Software delivery over the Web imposes a novel set of constraints on design. We outline the tradeoffs in this design space, motivate the choices necessary to deliver an application, and detail the lessons learned in the process. We discuss the role of Java and other Web technologies in the realization of the design. VPL facilitates the development and execution of parallel programs. The initial prototype supports high-level parallel programming based on Fortran 90 and High Performance Fortran (HPF), as well as explicit low-level programming with the MPI message-passing interface. Supplementary Java-based platform-independent tools for data and performance visualization are an integral part of the VPL. Pablo SDDF trace files generated by the Pablo performance instrumentation system are used for post-mortem performance visualization. © 1997 John Wiley & Sons, Ltd.

Algebraic computational models of OR-parallel execution of Prolog

A specification of the OR-parallel execution of Prolog programs, using CHOCS (calculus of higher order communicating systems) [24], is presented in the paper. A translation is defined from Prolog programs and goals to CHOCS processes: the execution of the CHOCS process corresponding to a goal mimics the OR-parallel execution of the original Prolog goal. In the translation, clauses and predicate definitions of a Prolog program correspond to processes. To model OR-parallelism, the processes \(P^1, \cdots , P^n\), corresponding to clauses \(C_1, \cdots , C_n\) (having the same head predicate \(p\)) start their execution concurrently, but, in order to respect the depth-first search rule, each \(P^i\) is guarded by the termination of the executions of processes \(P^j\)'s, \(j < i\). The computational model is proved correct with respect to the semantics of Prolog, as given in [4, 5]. Our model, because of its algebraic specification, can be easily used to prove properties of the parallel execution of Prolog programs. Moreover, the model exploits the maximum degree of parallelism, by giving the Prolog solutions in parallel, without any order among them. However, this model, being close to the Prolog semantics definition, contains sources of inefficiency which make it unpractical as a guide for the implementation. To overcome these problems, a new computational model is defined. This model is obtained by modifications of the basic one and thus its correctness can be easily proved. Finally, we show how to obtain models of different real implementations of OR-parallel Prolog by slight modification of the new model. The relations among all these models, in terms of parallelism degree, are studied by using the concepts of bisimulation and simulation, developed for concurrent calculi.

Stochastic bounds for parallel program execution times with processor constraints

A parallel program can be modeled as an acyclic directed graph, where a node represents a task, which is the smallest grain of computation to be assigned to a processor, and arcs stand for precedence (synchronization) constraints among the tasks. Due to different input data and unpredictable dynamic run time environments, the execution times of tasks as well as the entire program can be treated as random variables. In this paper, we develop some stochastic lower and upper bounds for parallel program execution times when there are limited processors. Such analysis can provide important information for job scheduling and resource allocation. For several typical classes of parallel programs, we derive very accurate closed form approximations for the bounds. Examples are also given to demonstrate the quality of the bounds derived.

Massively parallel execution of logic programs: A static approach

A static model for the parallel execution of logic programs on MIMD distributed memory systems is presented where a refutation is implemented through a process network returned by the compilation of the logic program. The model supports Restricted-AND, OR and stream parallelism and it is integrated with a set of static analyses to optimise the process network. Altogether, the processes interact according to a static data driven model with medium grain operators. Data flowing in the network is tagged to distinguish bindings belonging to the same refutation. A scheduling strategy to integrate low level scheduling and message flow control has been defined. Performance figures are presented.

Design and Simulation of the Aquarius-II Multiprocessor

Aquarius-II is a cache coherent multiprocessor system designed for the parallel execution of Prolog programs. It contains two tiers of memory: synchronization memory and high bandwidth (HB) memory. The synchronization memory consists of snooping caches connected to a bus and is used to store rendezvous points, synchronization bits, synchronization variables such as locks and semaphores and most of the write shared data. The HB memory is used to store the bulk of the application program code and data. It contains caches and an inexpensive VLSI chip based crossbar interconnection network to memory. The caches connected to the crossbar do not have full snooping capability. The architecture is evaluated by a full simulation of parallel execution of Prolog programs on Aquarius-II. The design details of the components of the architecture and simulation results are presented. Simulation results indicate that the two tier memory system significantly reduces memory interference and speeds up synchronization when compared to a single bus multi. This shared memory multiprocesor architecture has the potential to support other parallel programming paradigms.

Parallel execution of logic programs by load sharing

Execution of a logic program can be sped up by load sharing among a group of interconnected processors. Network contention and load imbalance are two potentially limiting factors that must be dealt with carefully. In this paper, we discuss a task scheduling scheme in which processors share the workload by voluntarily following a universal task distribution rule. Communication is reduced by having processors cooperate without frequent exchange of information. However, load balancing is rendered more difficult. We propose solutions to the problem by altering the shape of a search space to remove the so-called structural imbalance, and by following a statistically even task distribution rule. Simulation and experimental data indicate that the method is effective for a number of programs for which existing scheduling methods tend to generate overly fine-grained tasks that lead to heavy traffic in the network. Speed-up factors by the proposed technique are comparable to, or better than, that by a typical system which relies solely on dynamic task migration to balance the workload. The scheme appears to be particularly suitable for implementation on loosely coupled parallel platforms.

Improving granularity and locality of data in multiprocessor execution of functional programs

This paper discusses the major problems associated with the exploitation of locality of data structures in functional programs and how to improve it by controlling the granularity of such programs on a multiprocessor. Optimal execution of parallel programs depends on partitioning the program into modules (processes) and scheduling these modules for the shortest execution time possible. We show how various techniques can help us in increasing the granularity of parallel programs and enhancing the locality of its data structures on the larger-grain distributed multi-processor model, LAGER. Even in the presence of other granularity control methods, the overall parallel execution time of a parent process is minimized by scheduling its sub-processes in a particular order of their relative time complexities.

ANDES: Evaluating mapping strategies with synthetic programs

This paper presents the ANDES performance evaluation tool. ANDES is based on the synthetic execution of parallel programs and it is used for the evaluation of mapping strategies. The Meganode, a distributed memory parallel computer, is considered as our target architecture. ANDES takes into account a benchmark of quantitative models of parallel algorithms and a set of mapping strategies (greedy and iterative algorithms are used). We show how this tool allows an extensive comparison of mapping strategies by using the benchmark, the mapping strategies and different cost functions.

Using parallel program characteristics in dynamic processor allocation policies

Abstract In multiprocessors a parallel program's execution time is directly influenced by the number of processors it is allocated. The problem of scheduling parallel programs in a multiprogrammed environment becomes one of determining how to best allocate processors to the different simultaneously executing programs in order to minimize mean response time. In this paper we address the problem of how many processors to allocate to each of the executing parallel jobs by examining the following questions: 1. (1) Is allocating processors equally among all jobs (equipartitioning) a desirable property of a scheduling algorithm? 2. (2) Does using information about the service demand of parallel jobs significantly reduce mean response time? 3. (3) Does using information about the efficiency with which parallel jobs execute significantly reduce mean response time? 4. (4) Does allocating each job a number of processors corresponding to the knee of the execution time — efficiency curve significantly reduce mean response time? 5. (5) What are the desirable properties of a scheduler that is designed to minimize mean response time? The main contributions of this paper are: a first-order understanding of how processing power should be dynamically allocated to jobs, a new family of algorithms for dynamic processor allocation and a rough quantification of the benefits that may be realized by properly utilizing job characteristics when making processor allocation decisions. We believe that these new algorithms can be combined with recently demonstrated techniques for obtaining sufficiently accurate runtime estimates of job efficiencies to improve scheduler implementations for multiprogrammed multiprocessors.

Using parallel program characteristics in dynamic processor allocation policies

In multiprocessors a parallel program's execution time is directly influenced by the number of processors it is allocated. The problem of scheduling parallel programs in a multiprogrammed environment becomes one of determining how to best allocate processors to the different simultaneously executing programs in order to minimize mean response time. In this paper we address the problem of how many processors to allocate to each of the executing parallel jobs by examining the following questions: 1. (1) Is allocating processors equally among all jobs (equipartitioning) a desirable property of a scheduling algorithm? 2. (2) Does using information about the service demand of parallel jobs significantly reduce mean response time? 3. (3) Does using information about the efficiency with which parallel jobs execute significantly reduce mean response time? 4. (4) Does allocating each job a number of processors corresponding to the knee of the execution time — efficiency curve significantly reduce mean response time? 5. (5) What are the desirable properties of a scheduler that is designed to minimize mean response time? The main contributions of this paper are: a first-order understanding of how processing power should be dynamically allocated to jobs, a new family of algorithms for dynamic processor allocation and a rough quantification of the benefits that may be realized by properly utilizing job characteristics when making processor allocation decisions. We believe that these new algorithms can be combined with recently demonstrated techniques for obtaining sufficiently accurate runtime estimates of job efficiencies to improve scheduler implementations for multiprogrammed multiprocessors.

Performance evaluation of the slotted ring multiprocessor

As microprocessor speeds continue to improve at a very fast rate the bandwidth requirements for system level interconnections in multiprocessors may eventually rule out the use of shared buses even for small scale multiprocessors. On the other hand high speed unidirectional links are an emerging technology that has the potential to scale with microprocessor technology and could replace buses as the interconnection fabric for future multiprocessors. We evaluate the performance of the unidirectional slotted ring interconnection for small to medium scale shared memory systems, using a hybrid methodology of analytical models and trace driven simulations. We use memory traces from actual execution of parallel programs to drive detailed event driven simulations of a variety of ring and bus multiprocessors. Snooping and directory coherence protocols for the slotted ring are evaluated in the context of multitasking. Snooping is shown to outperform full map and linked list directory schemes in the unidirectional slotted ring, and it also compares favorably to high performance split transaction bus systems.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Evaluating the performance of a SISAL implementation of the abingdon cross image processing benchmark

There are many paradigms being promoted and explored for programming parallel computers, including modified sequential languages, new imperative languages and applicative languages. SISAL is an applicative language which has been designed by a consortium of industrial and research organizations for the specification and execution of parallel programs. It allows programs to be written with little concern for the structure of the underlying machine, thus the programmer is free to explore different ways of expressing the parallelism. A major problem with applicative languages has been their poor efficiency at handling large data structures. To counter this problem SISAL includes some advanced memory management techniques for reducing the amount of data copying that occurs. In this paper we discuss the implementation of some image processing benchmarks in SISAL and C to evaluate the effectiveness of the memory management code. In general, the SISAL program was easier to code than the C (augmented with the PARMACS macros) because we were not concerned with the parallel implementation details. We found that the SISAL performance was in general comparable to C, and that it could be brought in line with an efficient parallel C implementation by some programmer-specified code transformations.

A general harness for explicit parallel programming

In this paper a strategy for parallel programming is developed. The approach presented is based on separating and collecting the parallel control statements of a program into separate functions. This is called explicit parallel programming. This is done for two reasons. The first is that it allows the development of parallel control structures through natural usage, instead of forcing them on to the programmer; and the second is that such a separation makes it possible to develop a general harness for controlling the execution of parallel programs. An implementation of such a harness, called ExDaTa (standing for explicit data and task assignment), using a representative model for parallel processing is presented. The model is one where a parallel program is considered to be a collection of tasks with an order of execution imposed on them.