Parallel Execution Time Research Articles

Improving granularity and locality of data in multiprocessor execution of functional programs

This paper discusses the major problems associated with the exploitation of locality of data structures in functional programs and how to improve it by controlling the granularity of such programs on a multiprocessor. Optimal execution of parallel programs depends on partitioning the program into modules (processes) and scheduling these modules for the shortest execution time possible. We show how various techniques can help us in increasing the granularity of parallel programs and enhancing the locality of its data structures on the larger-grain distributed multi-processor model, LAGER. Even in the presence of other granularity control methods, the overall parallel execution time of a parent process is minimized by scheduling its sub-processes in a particular order of their relative time complexities.

Using parallel program characteristics in dynamic processor allocation policies

Abstract In multiprocessors a parallel program's execution time is directly influenced by the number of processors it is allocated. The problem of scheduling parallel programs in a multiprogrammed environment becomes one of determining how to best allocate processors to the different simultaneously executing programs in order to minimize mean response time. In this paper we address the problem of how many processors to allocate to each of the executing parallel jobs by examining the following questions: 1. (1) Is allocating processors equally among all jobs (equipartitioning) a desirable property of a scheduling algorithm? 2. (2) Does using information about the service demand of parallel jobs significantly reduce mean response time? 3. (3) Does using information about the efficiency with which parallel jobs execute significantly reduce mean response time? 4. (4) Does allocating each job a number of processors corresponding to the knee of the execution time — efficiency curve significantly reduce mean response time? 5. (5) What are the desirable properties of a scheduler that is designed to minimize mean response time? The main contributions of this paper are: a first-order understanding of how processing power should be dynamically allocated to jobs, a new family of algorithms for dynamic processor allocation and a rough quantification of the benefits that may be realized by properly utilizing job characteristics when making processor allocation decisions. We believe that these new algorithms can be combined with recently demonstrated techniques for obtaining sufficiently accurate runtime estimates of job efficiencies to improve scheduler implementations for multiprogrammed multiprocessors.

Using parallel program characteristics in dynamic processor allocation policies

In multiprocessors a parallel program's execution time is directly influenced by the number of processors it is allocated. The problem of scheduling parallel programs in a multiprogrammed environment becomes one of determining how to best allocate processors to the different simultaneously executing programs in order to minimize mean response time. In this paper we address the problem of how many processors to allocate to each of the executing parallel jobs by examining the following questions: 1. (1) Is allocating processors equally among all jobs (equipartitioning) a desirable property of a scheduling algorithm? 2. (2) Does using information about the service demand of parallel jobs significantly reduce mean response time? 3. (3) Does using information about the efficiency with which parallel jobs execute significantly reduce mean response time? 4. (4) Does allocating each job a number of processors corresponding to the knee of the execution time — efficiency curve significantly reduce mean response time? 5. (5) What are the desirable properties of a scheduler that is designed to minimize mean response time? The main contributions of this paper are: a first-order understanding of how processing power should be dynamically allocated to jobs, a new family of algorithms for dynamic processor allocation and a rough quantification of the benefits that may be realized by properly utilizing job characteristics when making processor allocation decisions. We believe that these new algorithms can be combined with recently demonstrated techniques for obtaining sufficiently accurate runtime estimates of job efficiencies to improve scheduler implementations for multiprogrammed multiprocessors.

Parallel Implementations of Iterated Runge-Kutta Methods

We investigate different parallel algorithms for the iter ated Runge-Kutta method on distributed memory mul tiprocessors for the solution of systems of ordinary differential equations (ODEs). The iterated Runge-Kutta method is an iteration scheme for the numerical solu tion of initial-value problems of nonstiff ODEs; embed ded approximation formulae are used to control the step size. The parallel algorithms realize potential par allelism across the method and the system in a group- SPMD (single-program, multiple-data) programming style, using an appropriate set of communication prim itives that can be implemented on all common topolo gies. A theoretical performance analysis with run-time formulae and a run-time simulation show the value of the algorithms. The implementation on the Intel iPSC/ 860 confirms the predicted run times. The speedup values depend strongly on the particular system of ODEs to be solved. The parallel iterated Runge-Kutta method is applied to a typical discretization problem, the discretized Brusselator equation. Application-spe cific modifications of the general parallel ODE solver are developed, which result in a considerable reduc tion in the parallel execution time.

Parallel event-driven logic simulation algorithms: tutorial and comparative evaluation

Parallel processing offers a viable way to improve the enormous execution time of the simulation of large VLSI designs. Various parallel logic simulation approaches have been proposed in recent years resulting in some ambiguity as to which scheme offers the best parallelism and execution time. To address these issues, the authors provide a detailed comparison of all four major types of event-driven logic simulation algorithms (synchronous, conservative asynchronous with deadlock avoidance, conservative asynchronous with deadlock detection and resolution, and optimistic asynchronous). The comparisons are carried out on an ideal parallel machine capable of extracting all available parallelism in a given algorithm. The simulation execution time, average parallelism and total messages required for a particular simulation algorithm are measured on the ISCAS combinational and sequential benchmark circuits. The use of an ideal parallel machine exposes characteristics of the simulation algorithms independent of the effects caused by particular parallel architectures or implementations. It is shown that a recently developed conservative asynchronous algorithm of the deadlock avoidance type and the optimistic asynchronous algorithm achieve the best parallel execution time results. However, the new conservative algorithm requires much less implementation overhead than the optimistic algorithm.

DISTRIBUTED COMPUTATION IN LOCAL AREA NETWORKS OF WORKSTATIONS

This paper describes the performance analysis of the local area network (LAN) of workstations for distributed computing applications. Several application programs represented in a task graph format are scheduled by the user workstation (master) for parallel execution in the idle or lightly loaded processors (slaves) in LAN. The partial results are collected by the master station to synchronize the operation of slave nodes. The system performance is characterized by the network speedup considering scheduling time, network load and size, communication time, TCP/IP communication overhead, task execution, and synchronization time. The sequential and parallel execution times and the performance degradation factors were measured during the implementation for various network loads. The measurement values and theoretical results have shown that the system performance is degraded mostly by the TCP/IP overhead (0.2 s), the network size, and the network load.

Performance Properties of Large Scale Parallel Systems

There are several metrics that characterize the performance of a parallel system, such as parallel execution time, speedup, and efficiency. A number of properties of these metrics have been studied. For example, it is a well known fact that given a parallel architecture and a problem of a fixed size, the speedup of a parallel algorithm does not continue to increase with increasing number of processors. It usually tends to saturate or peak at a certain limit. Thus, it may not be useful to employ more than an optimal number of processors for solving a problem on a parallel computer. This optimal number of processors depends on the problem size, the parallel algorithm, and the parallel architecture. In this paper we study the impact of parallel processing overheads and the degree of concurrency of a parallel algorithm on the optimal number of processors to be used when the criterion for optimality is minimization of the parallel execution time. We then study a more general criterion of optimality and show how operating at the optimal point is equivalent to operating at a unique value of efficiency that is characteristic of the criterion of optimality and the properties of the parallel system under study. We put the technical results derived in this paper in perspective with similar results that have appeared in the literature before and show how this paper generalizes and/or extends these earlier results.

A methodology for mapping and partitioning arbitraryn-dimensional nested loops into 2-dimensional VLSI arrays

A new methodology is proposed for mapping and partitioning arbitraryn-dimensional nested loop algorithms into 2-dimensional fixed size systolic arrays. Since planar VLSI arrays are easy to implement, our approach has good feasibility and applicability. In the transformation process of an algorithm, we take into account not only data dependencies imposed by the original algorithm but also space dependencies dictated by the algorithm transformation. Thus, any VLSI algorithm generated by our methodology has optimal parallel execution time and yet remains space-time conflict free. Moreover, a theory of the least complete set of interconnection matrices is proposed to reduce the computational complexity for finding all possible space transformations for a given algorithm.

Sorting algorithms on transputer arrays

In this paper, two sorting algorithms Key Select and Superquicksort are described and their performance is compared with other sorting algorithms namely Parallel Simple Merge, Partial Key Select and Pipemerge [6] on a transputer based message passing multiprocessor system. For each of these algorithms the variation of performance with the number of elements to be sorted and the number of processors is studied. The performance criteria chosen are parallel execution time, speedup and processing efficiency. Experimental and analytical results reveal that the algorithm Superquicksort gives the best performance for medium sized transputer based parallel processing systems, while for multiprocessors with more than sixteen processors, the algorithm Key Select is the fastest.

The influence of random delays on parallel execution times

Stochastic models are widely used for the performance evaluation of parallel programs and systems. The stochastic assumptions in such models exe intended to represent non-deterministic processing requirements as well as random delays due to inter-process communication end resource contention. In this paper, we provide compelling analytical and experimental evidence that in current and foreseeable shared-memory programs, communication delays introduce negligible variance into the execution time between synchronization points. Furthermore, we show using direct measurements of variance that other sources of randomness, particularly non-deterministic computational requirements, also do not introduce significant variance in many programs. We then use two examples to demonstrate the implications of these results for parallel program performance prediction models, as well as for general stochastic models of parallel systems.

A parallel algorithm for determining all eigenvalues of large real symmetric tridiagonal matrices

A method for determining all eigenvalues of large real symmetric tridiagonal matrices on multiprocessor system with vector facilities is presented. For finding the eigenvalues of a tridiagonal matrix, the method of the Sturm sequence is a standard method. The method uses bisection first to isolate all eigenvalues, bisection is and then to extract the eigenvalues to a predefined accuracy. For extracting the eigenvalues, bisection is accelerated by a superlinearly convergent zero finder, the Pegasus method. The evaluation of the Sturm sequence is the central component for both isolation and extraction. Some new ideas are presented, such as a method for weighting the values of the characteristics polynomial to avoid under- or overflow, a method for combining the Pegasus method with preceding bisection steps and a vectorization and parallelization strategy over intervals. The method was implemented and the results were measured on a SUPRENUM multiprocessor system with 16 processors and on a CRAY Y-MP8/832 with 8 processors. On the latter machine, both the sequential and parallel execution time of our algorithm ALLEV (ALL Eigen Values) presented in this paper are considerably shorter than the execution times of the vectorized EISPACK-routine TQL1 which uses the QL method.

Parallelism analyzers for parallel discrete event simulation

Discrete event simulation is usually time consuming. Recently, there has been a great deal of interest in using parallel computers to speed up the simulation process. Before the parallel simulation approach is applied, it is important to understand the inherent parallelism of simulation applications. A simple technique called critical path analysis was proposed to study paralllelism of simulation applications. This paper describes three critical path analysis algorithms based on different event-scheduling (process scheduling) policies. These algorithms are much simpler than a previous approach, where the events must be recorded in a trace and an extra pass is required to process the event trace. In our approach, the critical path analysis algorithms are integrated with the sequential simulation. At the end of the sequential simulation, the optimal parallel execution time is also computed. Livny proposed an algorithm similar to our approach (His, however, was designed for a specific language). Our algorithms can be integrated with sequential simulation programs written by users or be integrated with simulation languages. Another advantage of our algorithms over previous approaches is that ours can be used to study load balancing under different event-scheduling policies. Since our algorithms can be easily inserted in sequential simulation programs, critical path analysis can be applied to existing sequential programs without difficulty. The results can then be used to predict the performance of parallel simulation on similar applications. An example is given to show how useful information can be obtained from our algorithms.

PERFORMANCE ESTIMATION OF LU FACTORISATION ON MESSAGE PASSING MULTIPROCESSORS

The paper proposes an analytical model for estimating the performance of Pipelined Ring algorithm for LU factorisation on any distributed memory message passing multiprocessor. Expressions for parallel execution time and speedup are derived from the computation-communication characteristics of the algorithm. Earlier methods on performance estimation of LU factorisation have been based on determining the number of floating point operations in the best and worst cases. The methodology proposed in this paper follows a different approach and estimates the performance of LU factorisation from a measurement of the execution time of the algorithm on a single processor and from a knowledge of the number of bytes communicated in different steps of the algorithm. The expression for parallel execution time of LU factorisation derived from the analytical model has been validated with experimental values obtained 011 a sixty-four transputer based multiprocessor. Results indicate that the methodology proposed in this paper can be used for estimating the execution time of Pipelined Ring algorithm for LU factorisation on any distributed memory message passing multiprocessor with high accuracy.

Efficient processor assignment algorithms and loop transformations for executing nested parallel loops on multiprocessors

An important issue for the efficient use of multiprocessor systems is the assignment of parallel processors to nested parallel loops. It is desirable for a processor assignment algorithm to be fast and always generate an optimal processor assignment. The paper proposes two efficient algorithms to decide the optimal number of processors assigned to each individual loop. Efficient parallel counterparts of these two algorithms are also presented. These algorithms not only always generate an optimal processor assignment, but also are much faster than the exiting optimal algorithm in the literature. The paper discusses improving the performance of parallel execution by transforming a nested parallel loop into a semantically equivalent one. Three loop transformations are investigated. It is observed that, in most cases, the parallel execution time is improved after applying these transformations.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Toward a better parallel performance metric

The traditional definition of ‘speedup’ as the ratio of sequential execution time to parallel execution time has been widely accepted. One drawback to this metric is that it tends to reward slower processors and inefficient compilation with higher speedup. It seems unfair that the goals of high speed and high speedup are at odds with each other. In this paper, the ‘fairness’ of parallel performance metrics is studied. Theoretical and experimental results show that the most commonly used performance metric, parallel speedup, is ‘unfair’, in that it favors slow processors and poorly coded programs. Two new performance metrics are introduced. The first one, sizeup, provides a ‘fair’ performance measurement. The second one is a generalization of speedup — the generalized speedup, which recognizes that speedup is the ratio of speeds, not times. The relation between sizeup, speedup, and generalized speedup are studied. The various metrics have been tested using a real application that runs on an nCUBE 2 multicomputer. The experimental results closely match the analytical results.

Automatic partitioning of a program dependence graph into parallel tasks

In this paper, we describe a general interprocedural framework for partitioning a program dependence graph into parallel tasks for execution on a multiprocessor system. Partitioning techniques are necessary to execute a parallel program at the appropriate granularity for a given target multiprocessor. The problem is to determine the best trade-off between parallelism and overhead. It is desirable for the partitioning to be performed automatically, so that the programmer can write a parallel program without being burdened by details of the overhead target multiprocessor, and so that the same parallel program can be made to execute efficiently on different multiprocessors. For each procedure, the partitioning algorithm attempts to minimize the estimated parallel execution time. The estimated parallel execution time reflects a trade-off between parallelism and overhead and is minimized at an optimal intermediate granularity of parallelism. Execution-profiling information is used to obtain accurate execution-time estimates. The partitioning framework has been completely implemented in the PTRAN system at the IBM Thomas J. Watson Research Center. Partitioned parallel programs generated by this prototype system have been executed on the IBM 3090™ and RP3 multiprocessor systems.

Path planning on a ring of processors∗∗

In this paper, we discuss the implementation of Bitz and Kung's path planning algorithm on a ring of general-purpose processors. We show that Bitz and Kung's algorithm, originally designed for the Warp machine, is not efficient in this context, due to the intensive inter-processor communications that it requires. We design a modified version that is much more performing. The new version updates a segment of k positions within a step and allocates blocks of r consecutive rows of the map to the processors in a wraparound fashion. Bitz and Kung's algorithm corresponds to the situation (k, r) = (l, 1). We analytically determine the optimal values of the parameters (k, r) which minimize the parallel execution time as a function of the problem size n and of the number of processors p. The theoretical results are nicely corroborated by numerical experiments on a ring of 32 transputers.

The impact of domain partitioning on the performance of a shared memory multiprocessor

The impact of the choice of domain partition/stencil pairs in solving elliptic partial differential equations on a multiprocessor system with both global shared memory and locl memory is considered. A parallel execution time model is given for square and hexagonal partitions using both five-point and nine-point star discretizations. Results from the model for a particular multiprocessor system show that keeping copies of partition boundaries in local memory significantly improves performance.

Some issues in parallel processing as encountered on the Denelcor HEP

We present a collection of differing parallel implementation of a single, computationally intensive algorithm that models the collisionless, electrostatic interaction between two relatively moving plasma beams. This numerical simulation uses a method, important in many scientific applications known as the Particle-in-Cell (PIC) method. Our aim in this study is to determine the advantages and disadvantages associated with those various parallel implementations. Our experiments with parallelizing this particular numerical simulation, referred to throughout this paper as the PIC code, were performed on a single Denelcor HEP Process Execution Module (PEM). A complete set of parallel processing speedups and execution times as a function of number of processes is presented.

Parallel, iterative solution of sparse linear systems: Models and architectures

Solving large, sparse, linear systems of equations is a fundamental problems in large scale scientific and engineering computation. A model of a general class of asynchronous, iterative solution methods for linear systems is developed. In the model, the system is solved by creating several cooperating tasks that each compute a portion of the solution vector. A data transfer model predicting both the probability that data must be transferred between two tasks and the amount of data to be transferred is presented. This model is used to derive an execution time model for predicting parallel execution time and an optimal number of tasks given the dimension and sparsity of the coefficient matrix and the costs of computation, synchronization, and communication. The suitability of different parallel architectures for solving randomly sparse linear systems is discussed. Based on the complexity of task scheduling, one parallel architecture, based on a broadcast bus, is presented and analyzed.