Execution Of Parallel Programs Research Articles

A quasi-optimal cluster allocation strategy for parallel program execution in distributed systems using genetic algorithms

This paper shows an approach to find quasi-optimal solutions to the first optimization stage of parallel program tasks allocation problem, in an internet distributed system. The user initiates program execution from an arbitrary node in an arbitrary cluster, the parallel tasks comprising the program migrate to quasi-optimal clusters using an strategy that tries to minimize intercluster traffic of the parallel program execution.Genetic algorithms (GAs) [11] are used to provide a set of timely, quasi-optimal solutions.

Partitioning and mapping of nested loops for linear array multicomputers

In distributed-memory multicomputers, minimizing interprocessor communication is the key to the efficient execution of parallel programs. In order to reduce the amount of communication overhead, parallel programs on multicomputers must be carefully scheduled by parallelizing compilers. This paper proposes some compilation techniques for partitioning and mapping nested loops with constant data dependences onto linear array multicomputers. First, a systematic partition strategy is proposed to project ann-dimensional computational structure, representing ann-nested loop, onto a line to form a one-dimensional projected structure with low communication overhead. Then, a mapping algorithm is proposed for mapping the partitioned loops onto linear arrays in a way that balances the workload and minimizes the communication cost among processors. Finally, parallel execution codes can be automatically generated for such linear array multicomputers.

Accurate Predictions of Parallel Program Execution Time

It has been difficult to develop simple formulations to predict the execution time of parallel programs due to the complexity of characterizing parallel hardware and software. In an attempt to clarify these characterizations, we introduce a methodology for applying a simple performance model based on Amdahl′s law. Our formulation results in accurate predictions of execution time on available systems, allowing programmers to select the optimal number of processors to apply to a particular problem or to select an appropriate problem size for the number of processors available. In short, we accurately quantify the scalability of a specific algorithm when it is run on a specific parallel computer. Our predictions are based on simple experiments that characterize machine performance and on a simple analysis of the parallel program. We illustrate our method for a program executed on a Sequent Symmetry multiprocessor with 20 processors. Our predictions closely match experimental results, differing by no more than 5% from the actual execution times. Our results illustrate key performance limitations of parallel systems, showing the impact of overhead and the scaling of problem size.

The communication software and parallel environment of the IBM SP2

This paper describes the software available on the IBM SP2™ for parallel program development and execution. It presents the rationale for the design of the Message-Passing Library used on the SP2, outlines its current implementation, and gives information on performance. In addition, the paper describes the programming environment and the program development tools available for developing and executing parallel codes.

Using true concurrency to model execution of parallel programs

Parallel execution of a programR (intuitively regarded as a partial order) is usually modeled by sequentially executing one of the total orders (interleavings) into which it can be embedded. Our work deviates from this serialization principle by usingtrue concurrency to model parallel execution. True concurrency is represented via completions ofR tosemi total orders, called time diagrams. These orders are characterized via a set of conditions (denoted byCt), yielding orders or time diagrams which preserve some degree of the intended parallelism inR. Another way to express semi total orders is to use re-writing or derivation rules (denoted byCx) which for any programR generates a set of semi-total orders. This paper includes a classification of parallel execution into three classes according to three different types ofCt conditions. For each class a suitableCx is found and a proof of equivalence between the set of all time diagrams satisfyingCt and the set of all terminalCx derivations ofR is devised. This equivalence between time diagram conditions and derivation rules is used to define a novel notion of correctness for parallel programs. This notion is demonstrated by showing that a specific asynchronous program enforces synchronous execution, which always halts, showing that true concurrency can be useful in the context of parallel program verification.

Data and Task Alignment in Distributed-Memory Architectures

Different alignments of multidimensional arrays on a meshconnected SIMD architecture result in different communication patterns during parallel program execution. In this paper a compile-time selection of data alignment that minimizes communication cost is discussed. First, it is shown that the selection is computationally as hard as a subclass of the well-known problem of finding the closest vector in a lattice. The NP-hardness of the latter problem is proven. Then, two algorithms for exact minimum solution(s) are discussed. Although the complexity of these algorithms is exponential, for small lattices often generated by parallel scientific computation the execution times of these algorithms may be acceptable. A polynomial-time algorithm for finding an approximate solution is also described. Finally, improvement in communication cost resulting from alignments for randomly generated graphs are presented.

Machine independent AND and OR parallel execution of logic programs. II. Compiled execution

For pt.I. see ibid., p. 170-80. In pt.I, we presented a binding environment for the AND and OR parallel execution of logic programs. This environment was instrumental in rendering a compiler for the AND and OR parallel execution of logic programs machine independent. In this paper, we describe a compiler based on the Reduce-OR process model (ROPM) for the parallel execution of Prolog programs, and provide performance of the compiler on five parallel machines: the Encore Multimax, the Sequent Symmetry, the NCUBE 2, the Intel i860 hypercube and a network of Sun workstations. The compiler is part of a machine independent parallel Prolog development system built on top of a run time environment for parallel programming called the Chare kernel, and runs unchanged on these multiprocessors. In keeping with the objectives behind the ROPM, the compiler supports both on and independent AND parallelism in Prolog programs and is suitable for execution on both shared and nonshared memory machines. We discuss the performance of the Prolog compiler in some detail and describe how grain size can be used to deliver performance that is within 10% of the underlying sequential Prolog compiler on one processor, and scale linearly with increasing number of processors on problems exhibiting sufficient parallelism. The loose coupling between parallel and sequential components makes it possible to use the best available sequential compiler as the sequential component of our compiler.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Analysis of Asynchronous Execution Streams with I-Caching in Massively Parallel Systems

Real world parallel systems on a massive scale have been limited to SIMD architectures due to the prohibitive cost of multiple instruction memories. Although many of these systems could benefit from an asynchronous MIMD control structure, it is unlikely that every processing element will execute a different set of instructions. A novel instruction caching system is presented that takes advantage of this to permit a machine to operate in an asynchronous SPMD mode without storing multiple copies of the program. An analytical model is constructed to represent parallel program execution streams and used to drive the instruction caches. The model′s predictions are supported by multiprocessor simulation data. The results are used to examine the parameters affecting caching performance. Information gained from this study will be used to direct future research.

An adaptive granularity control algorithm for the parallel execution of functional programs

An on-line (run time), adaptive algorithm for granularity control of the parallel execution of functional programs is presented. This algorithm utilizes the inherent parallelism of functional programs, yet it endeavours to limit excessive parallelism. It can be applied to programs without any prior knowledge of their behaviour, and it does not require the programmer to add explicit parallel annotations. The algorithm takes advantage of the ability to expose, at run time, the shape of the underlying computation graph of the program. It also balances between the amount of local computation and the costs of remote execution. Based on these considerations, the algorithm allocates tasks to processing elements in a way that tries to minimize the overall execution time of the program. Continuous sampling of the computation graph allows self-tuning of the algorithm, when the underlying graph changes gradually.

Optimized parallel execution of declarative programs on distributed memory multiprocessors

In this paper, we focus on the compiling implementation of parallel logic language PARLOG and functional language ML on distributed memory multiprocessors. Under the graph rewriting framework, a Heterogeneous Parallel Graph Rewriting Execution Model (HPGREM) is presented firstly. Then based on HPGREM, a parallel abstract machine PAM/TGR is described. Furthermore, several optimizing compilation schemes for executing declarative programs on transputer array are proposed. The performance statistics on a transputer array demonstrate the effectiveness of our model, parallel abstract machine, optimizing compilation strategies and compiler.

What to Draw? When to Draw? An Essay on Parallel Program Visualization

Parallel programs are difficult to understand and the research community is determined to provide tools to help improve that understanding. As a result, there is intense interest in using visualization to understand the execution of parallel programs. But visualization is like the surgeon’s knife: properly used it will quickly cut to the essentials; improper use brings colorful but disastrous results. The results from a visualization are almost always visually appealing, but I claim that they only occasionally are useful in debugging a program or improving its performance.

Visualization of Message Passing Parallel Programs with the TOPSYS Parallel Programming Environment

Parallel programming is orders of magnitudes more complex than writing sequential programs. This is particularly true for programming distributed memory multiprocessor architectures based on message passing programming models. Apart from understanding the sequential parts of the parallel program, new degrees of freedom lead to additional problems. Understanding the synchronization and communication behavior of parallel programs is the most critical issue in programming distributed memory multiprocessors. The paper describes methods and tools for visualization and animation of the dynamic execution of parallel programs. Based on an evaluation and classification of existing visualization environments, the visualization and animation tool VISTOP (VISualization TOol for Parallel Systems) is presented as part of the integrated tool environment TOPSY S (TOols for Parallel SYStems) for programming distributed memory multiprocessors. VISTOP supports the interactive on-line visualization of message passing programs based on various views; in particular, a process graph based concurrency view for detecting synchronization and communication bugs.

An iteration partition approach for cache or local memory thrashing on parallel processing

Parallel processing systems with cache or local memory in the memory hierarchies are considered. These systems have a local cache memory in each processor and usually employ a write-invalidate protocol for the cache coherence. In such systems, a problem called 'cache or local memory thrashing' can arise in executions of parallel programs, when the data unnecessarily moves back and forth between the caches or local memories in different processors. An approach to eliminate, or at least to reduce, such movement for nested parallel loops is presented. It is based on relations between array element accesses and enclosed loop indexes in the loops. The relations can be used to assign processors to execute the appropriate iterations for parallel loops in the loop nests with respect to the data in their caches or local memories. An algorithm for calculating the correct iteration of the parallel loop in terms of loop indexes of the previous iterations executed in the processor is presented. This method benefits parallel code with nested loop structures in a wide range of applications. The experimental results show that the technique can achieve speedups up to 2.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Heuristic Methods for Dynamic Load Balancing in a Message-Passing Multicomputer

In this paper, a new adaptive scheme is presented for dynamic load balancing in a message-passing multicomputer. The scheme is based on using easy-to-implement heuristics and adaptive threshold in balancing the system load among dispersed nodes. It uses a distributed control over all computer nodes as coordinated by an information collector. Four heuristic methods are presented here, which are distinguished by the ranges for location and threshold update policies and by the disciplines used for determining the load transfer destination. A parallel simulator with distributed load balancers is developed on an iPSC/2 hypercube multi-computer. The load balancing scheme is evaluated on the basis of the effects of system utilization, load imbalance, communication and migration overhead, and multicomputer size. Relative merits of the four methods are revealed under various physical configurations of the multicomputer. The application potentials are discussed for parallel execution of AI-oriented programs and distributed semantic network data bases.

The ParaScope parallel programming environment

The ParaScope parallel programming environment, developed to support scientific programming of shared-memory multiprocessors, is described. It includes a collection of tools that use global program analysis to help users develop and debug parallel programs. The focus is on ParaScope's compilation system. The compilation system extends the traditional single-procedure compiler by providing a mechanism for managing the compilation of complete programs. The ParaScope editor brings both compiler analysis and user expertise to bear on program parallelization. The debugging system detects and reports timing-dependent errors, called data races, in execution of parallel programs. A project aimed at extending ParaScope to support programming in FORTRAN D, a machine-independent parallel programming language for use with both distributed-memory and shared-memory parallel computers, is described.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Evaluation of the continuation bit in the cyclic pipeline computer

The Cyclic Pipeline Computer (CPC) is a shared resource computer which shares its pipeline among multiple instruction streams to create distinct virtual processors. This effectively removes data dependencies which reduce the performance of highly pipelined computers. However, when executing unbalanced parallel programs, the CPC suffers from performance degradation due to synchronization overhead. In this paper we introduce a hardware extension called the continuation bit. The continuation bit controls the instruction issue and determines whether an instruction is issued from the same stream or the next stream. Simulations of various programs indicate that the continuation bit is able to balance the execution of parallel programs by exploiting instruction level parallelism.

An abstract prolog machine based on parallel resolution principle

We describe an abstract machine, called SPUM, for the AND parallel execution of Prolog programs. We also present a resolution algorithm for logic programs, on which the SPUM architecture is based. The algorithm, called SPU, is based on parallel resolution principle and allows parallel execution of unificationsof a deterministic path of the proof tree, giving in effect dependent and-parallelism. We also present preliminary results which are indicative of the performance expected from SPUM.

A methodology for performance analysis of parallel computations with looping constructs

This paper describes a novel modeling methodology for evaluating the execution of parallel programs containing looping constructs. Our approach employs analytic methods to obtain an estimate of the average execution time of such a program in a distributed, multiple-computer environment. These tools are based on an amalgamation of queuing network theory (used to model physical system characteristics) and graph models of program behavior. To evaluate a complex program, we first decompose it into a number of program segments with “simpler” structures; then analyze each segment independently; and, finally, combine individual results to produce an approximate solution for the whole program. A realistic example is used to illustrate the application of this performance evaluation methodology.

Computer-assisted microanalysis of parallel programs

This paper consists of two parts: the first provides the theoretical foundations for analyzing parallel programs and illustrates how the theory can be applied to estimate the execution time of a class of parallel programs being executed on a MIMD computer. The second part describes a program analysis system, based on the theoretical model, which allows a user to interactively analyze the results of executing (or simulating the execution) of such parallel programs. Several examples illustrating the use of the tool are presented. A novel contribution is the separation (both at the conceptual and the implementation levels) of the machine-independent and the machine-dependent parts of the analysis. This separation enables the users of the system to establish speed-up curves for machines having varying characteristics.

Parallel programming with data structures and higher order functions

Linear lists, which are the standard data structure in functional programming languages, have proved to be useful for many applications. Yet for several reasons it seems that there is a need for additional applicative data structures: Firstly, functional programming languages are well suited for parallel execution, because programs written in these languages cannot have side effects, and expressions can be evaluated independently. But parallel execution of functional programs is often hampered by the linear structure of lists, which supports only one element at a time algorithms. Secondly, lists lack the expressive power that we need in general purpose languages. For example, there is a need for array-like data structures. Thirdly, many functions on lists have to be defined recursively. No matter how efficiently recursion may be implemented, it will always fall short of the efficiency that can be obtained with imperative implementations. We propose a choice of data structures and corresponding higher order functions that enable programmers to write concise and (almost) recursion-free functional programs which have a higher degree of potential parallelism. Possible implementation techniques are discussed, and the major complexity result is that, under parallel evaluation, many functions on our data structures need only logarithmic time, whereas using lists results in linear time complexity.