Parallel Programming System Research Articles

Evaluating design alternatives for reliable communication on high-speed networks

We systematically evaluate the performance of five implementations of a single, user-level communication interface. Each implementation makes different architectural assumptions about the reliability of the network hardware and the capabilities of the network interface. The implementations differ accordingly in their division of protocol tasks between host software, network-interface firmware, and network hardware. Using microbenchmarks, parallel-programming systems, and parallel applications, we assess the performance impact of different protocol decompositions. We show how moving protocol tasks to a relatively slow network interface yields both performance advantages and disadvantages, depending on the characteristics of the application and the underlying parallel-programming system. In particular, we show that a communication system that assumes highly reliable network hardware and that uses network-interface support to process multicast traffic performs best for all applications.

Evaluating design alternatives for reliable communication on high-speed networks

We systematically evaluate the performance of five implementations of a single, user-level communication interface. Each implementation makes different architectural assumptions about the reliability of the network hardware and the capabilities of the network interface. The implementations differ accordingly in their division of protocol tasks between host software, network-interface firmware, and network hardware. Using microbenchmarks, parallel-programming systems, and parallel applications, we assess the performance impact of different protocol decompositions. We show how moving protocol tasks to a relatively slow network interface yields both performance advantages and disadvantages, depending on the characteristics of the application and the underlying parallel-programming system. In particular, we show that a communication system that assumes highly reliable network hardware and that uses network-interface support to process multicast traffic performs best for all applications.

Evaluating design alternatives for reliable communication on high-speed networks

We systematically evaluate the performance of five implementations of a single, user-level communication interface. Each implementation makes different architectural assumptions about the reliability of the network hardware and the capabilities of the network interface. The implementations differ accordingly in their division of protocol tasks between host software, network-interface firmware, and network hardware. Using microbenchmarks, parallel-programming systems, and parallel applications, we assess the performance impact of different protocol decompositions. We show how moving protocol tasks to a relatively slow network interface yields both performance advantages and disadvantages, depending on the characteristics of the application and the underlying parallel-programming system. In particular, we show that a communication system that assumes highly reliable network hardware and that uses network-interface support to process multicast traffic performs best for all applications.

Parallel Logic Programming Systems on Scalable Architectures

Parallel logic programming (PLP) systems are sophisticated examples of symbolic computing systems. PLP systems address problems such as allocating dynamic memory, scheduling irregular computations, and managing different types of implicit parallelism. Most PLP systems have been developed for bus-based architectures. However, the complexity of PLP systems and the large amount of data they process raise the question of whether logic programming systems can still achieve good performance on modern scalable architectures, such as distributed shared-memory (DSM) systems. In this work we use execution-driven simulation of a cache-coherent DSM architecture to investigate the performance of Andorra-I, a state-of-the-art PLP system, on a modern multiprocessor. The results of this simulation show that Andorra-I exhibits reasonable running time performance, but it does not scale well. Our detailed analysis of cache misses and their sources expose several opportunities for improvements in Andorra-I. Based on this analysis, we modify Andorra-I using a set of simple techniques that led to significantly better running time and scalability. These results suggest that Andorra-I can and should perform well on modern multiprocessors. Furthermore, as Andorra-I shares its main data structures with several PLP systems, we conclude that the methodology and techniques used in our work can greatly benefit these other PLP systems.

Preface: Volume 30, Issue 3

One of the main areas of research in logic programming is the design and implementation of sequential and parallel (constraint) logic programming systems. This research goes broadly from the design and specification of novel implementation technology to its actual evaluation in real life situations. A series of workshops on Implementations of Logic Programming Systems, previously held in Budapest (1993), Ithaca (1994), Portland (1995), Bonn (1996), Port Jefferson (1997), Manchester (1998) and Las Cruces (1999) provided a forum for ongoing research on the design and implementation of sequential and parallel (constraint) logic programming systems.This volume contains a collection of papers presented at the Workshop on Parallelism and Implementation Technology for (Constraint) Logic Programming, held in Las Cruces on December 1st, 1999, in conjunction with ICLP'99. The workshop was sponsored and organised by COMPULOG AMERICAS. The workshop also received support from the Association for Logic Programming and from the Department of Computer Science, New Mexico State University.Papers from both academia and industry were invited. Preference was given to the analysis and description of implemented systems (or currently under implementation) and their associated techniques, problems found in their development or design, and steps taken towards the solution of these problems.Topics included, but were not limited to: •standard and non—standard sequential implementation schemes (e.g., generalization/modification of WAM, translation to C, etc.);implementation of parallel logic programming systems;balance between compile-time effort and run-time machinery;techniques for the implementation of different declarative programming paradigms based on, or extending, logic programming (e.g., constraint logic programming, concurrent constraint languages, equational-logic languages);performance evaluation of sequential and parallel logic programming systems, both through benchmarking and using real world applications;other implementation-related issues, such as memory management, register allocation, use of global optimisations, etc.We were very fortunate to have so many interesting research papers, ranging over widely different subjects and giving a broad coverage of current research in sequential and parallel implementation of logic programming systems. Papers on sequential logic programming systems, focus on varied topics: constraint evaluation, support for extensions to logic programming, and abstract machines for performance evaluation. Papers on parallel logic programming systems also focus on diverse topics ranging from distributed implementations, garbage collection, to optimisations for exploiting and-or parallelism.The editors would like to thank all authors that chose to submit their work to this book, and also for their cooperation in making this document possible. We would also like to thank all referees involved in assessing the papers in this special volume.This volume will be published as volume 30, Issue 3 in the series Electronic Notes in Theoretical Computer Science (ENTCS). This series is published electronically through the facilities of Elsevier Science B.V. and its auspices. The volumes in the ENTCS series can be accessed at the URL http://www.elsevier.nl/locate/entcsMarch 14, 2000Horst Reichel

The parallel logic programming system in the FGCS project and its future directions

In the fifth generation computer systems (FGCS) project, a parallel logic programming language, KL1, was adopted as the project's kernel language. It was not only used to determine architectures of highly parallel machines called parallel inference machines (PIMs) consisting of about 1000 element processors but also used as a system description language to develop basic software such as a parallel operating system (PIMOS), and symbolic processing and knowledge processing application systems such as knowledge description languages, a parallel theorem prover, and a protein sequence analysis program. It achieved great success in exploiting of parallelism involved in several important application systems. The prototype of the FGCS attained a linear speed-up that was proportional to the number of processing elements (PEs) for the application systems we had targeted. The MGTP parallel theorem prover was one of such application systems, and can prove theorems based on full first-order logic. Thus, it indicates the possibility of designing a new practical knowledge representation language whose expressive power will be much greater than that of conventional ones. In the FGCS follow-on project, KL1 and its programming system were ported to Unix-based stock parallel machines. This new system called KLIC is expected to greatly extend the use of highly parallel systems.

Generating parallel programs from skeleton based specifications

The Static Iterative Transformation Specification System (SITSS) is a portable parallel programming system. This paper describes the techniques used in SITSS to generate parallel programs from user specifications of algorithms. The paper first outlines SITSS, the algorithmic skeleton on which SITSS is based, and the user interface. The techniques for analysing the specification selecting a data partitioning strategy, and generating parallel code are described. A comparison of SITSS produced code against hand-written code is then presented, followed by a discussion of areas for future work to enhance the system.

A parallel implementation of genetic programming that achieves super-linear performance

This paper describes the successful parallel implementation of genetic programming on a network of processing nodes using the transputer architecture. With this approach, researchers of genetic algorithms and genetic programming can acquire computing power that is intermediate between the power of currently available workstations and that of supercomputers at intermediate cost. This approach is illustrated by a comparison of the computational effort required to solve a benchmark problem. Because of the decoupled character of genetic programming, our approach achieved a nearly linear speed up from parallelization. In addition, for the best choice of parameters tested, the use of subpopulations delivered a super-linear speed-up in terms of the ability of the algorithm to solve the problem. Several examples are also presented where the parallel genetic programming system evolved solutions that are competitive with human performance.

Experience with parallel programming using code templates

For almost a decade we have been working at developing and using template-based models for parallel computing. Template-based models separate the specification of the parallel structuring aspects from the application code that is to be parallelized. A user provides the application code and specifies the parallel structure of the application using high-level icons, called templates. The parallel programming system then generates the code necessary for parallelizing the application. The goal here is to provide a mechanism for quick and reliable development of coarse-grain parallel applications that employ frequently occurring parallel structures. Our initial template-based system, FrameWorks, was positively received but had a number of shortcomings. The Enterprise parallel programming environment evolved out of this work. Now, after several years of experience with the system, its shortcomings are becoming evident. Controlled experiments have been conducted to assess the usability of our system in comparison with other systems. The paper outlines our experiences in developing and using these systems. A list of desirable characteristics of template-based models is given. The FrameWorks and Enterprise systems are discussed in the context of these characteristics and the results of our usability experiments. Many of our observations are relevant to other parallel programming systems, even though they may be based on different assumptions. Although template-base models have the potential for simplifying the complexities of parallel programming, they have yet to realize these expectations for high-performance applications. © 1998 John Wiley & Sons, Ltd.

Experience with parallel programming using code templates

For almost a decade we have been working at developing and using template-based models for parallel computing. Template-based models separate the specification of the parallel structuring aspects from the application code that is to be parallelized. A user provides the application code and specifies the parallel structure of the application using high-level icons, called templates. The parallel programming system then generates the code necessary for parallelizing the application. The goal here is to provide a mechanism for quick and reliable development of coarse-grain parallel applications that employ frequently occurring parallel structures. Our initial template-based system, FrameWorks, was positively received but had a number of shortcomings. The Enterprise parallel programming environment evolved out of this work. Now, after several years of experience with the system, its shortcomings are becoming evident. Controlled experiments have been conducted to assess the usability of our system in comparison with other systems. The paper outlines our experiences in developing and using these systems. A list of desirable characteristics of template-based models is given. The FrameWorks and Enterprise systems are discussed in the context of these characteristics and the results of our usability experiments. Many of our observations are relevant to other parallel programming systems, even though they may be based on different assumptions. Although template-base models have the potential for simplifying the complexities of parallel programming, they have yet to realize these expectations for high-performance applications. © 1998 John Wiley & Sons, Ltd.

Software Tools for Scalable Multilevel Application Engineering

Contemporary activities at CSCS/SCSC have resulted in two complementary software systems for practical parallel programming. Both developments are user oriented and application driven, efficiently exploiting and reusing dem onstrated portable technologies at multiple levels that we have proven to scale to systems with large numbers of processors. The Annai parallel application engineering environment supports existing standards for portable pro gram development (HPF, Fortran, C, MPI) and offers con venient program browsing and navigation, execution con trol, and interaction mechanisms. Specific functionality for parallel programming includes high-level language sup port for unstructured computations, interactive source- level symbolic debugging with deadlock detection and deterministic execution replay, SPMD/data-parallel debug ging with distributed breakpoints and array visualization, scalable profile summary displays of execution statistics accumulated at runtime, and detailed program evolution and processor interaction charts. The intelligent program development environment (PDE) is typified by four char acteristics supporting programming at a very abstract level, closer to the scientist's perspective: application- oriented problem description formalisms, the use of design skeletons and templates, an interactive user guidance mechanism, and automatic program synthesis tech niques. Ongoing work aims at the integration and further development of sofar demonstrated results with additional advanced technologies into comprehensive application engineering and problem-solving environments for pro ductive parallel computing with distributed resources.

Quantifying the Performance Differences between PVM and TreadMarks

This paper compares two systems for parallel programming on networks of workstations: Parallel Virtual Machine (PVM), a message-passing system, and TreadMarks, a software distributed shared-memory (DSM) system. The eight applications used in this comparison are Water and Barnes–Hut from the SPLASH benchmark suite; 3-D FFT, Integer Sort (IS), and Embarrassingly Parallel (EP) from the NAS benchmarks; ILINK, a widely used genetic linkage analysis program; and Successive Over-Relaxation (SOR) and Traveling Salesman (TSP). Two different input data sets are used for five of the applications. We use two execution environments. The first is a 155 Mbps ATM network with eight Sparc-20 model 61 workstations; the second is an eight-processor IBM SP/2. The differences in speedup between TreadMarks and PVM depend mostly on the applications, and only to a much lesser extent on the platform and the data set used. In particular, the TreadMarks speedup for six of the eight applications is within 15% of that achieved with PVM. For one application, the difference in speedup is between 15% and 30%, and for another, the difference is around 50%. We identified four important factors that contribute to the lower performance of TreadMarks: (1) extra messages due to the separation of synchronization and data transfer, (2) extra messages to handle access misses caused by the use of an invalidate protocol, (3) false sharing, and (4) diff accumulation for migratory data. We have quantified the effects of the last three factors by measuring the performance gain when each is eliminated. Of the three factors, TreadMarks' use of a separate request message per page of data accessed is the most important. The effect of false sharing is comparatively low. Reducing diff accumulation benefits migratory data only when the diffs completely overlap. When these performance impediments are removed, all of the TreadMarks programs perform within 25% of PVM, and for six out of eight experiments, TreadMarks is less than 5% slower than PVM.

Thread- and Process-based Implementations of the pSystem Parallel Programming Environment

Run-time work distribution in parallel programming systems is usually accomplished through the use of dynamic scheduling heuristics. Their sensitivity to run-time information such as global work-load, task granularity, data dependencies, locality of information, among others, is essential when trying to optimize performance. Adaptive schedulers that base their decisions on feed-back from the system are therefore of special importance. We have developed and used a general purpose parallel programming system, the pSystem, that also served as a test-bed environment on which we have experimented and studied the performance of distinct scheduling heuristics. Currently, we have two versions of the system: one based on Unix processes; and the other on Solaris threads. Threads (particularly user-level threads) are usually associated with low execution overheads, since they require minimal interaction with the operating system kernel. This suggests that lower grain parallelism may be more effectively exploited with a thread-based parallel programming system. Performance analysis of both implementations over a set of well known benchmarks, with various schedulers, shows that threads scale better under higher system loads and/or when the granularity of the tasks being executed is below a given threshold value. This paper starts with a description of the design and implementation of the pSystem computational model, followed by a detailed description of several experiments and the analysis of their results. © 1997 John Wiley & Sons, Ltd.

Parallel Implementation of Tree Skeletons

Trees are a useful data type, but they are not routinely included in parallel programming systems, in part because their irregular structure makes partitioning and scheduling difficult. We present a method for algebraically constructing implementations of tree skeletons, high-level homomorphic operations that execute in parallel. Many computations on binary trees can be performed inO(logn) parallel time usingnprocessors, even taking account of communication costs. We extend these results to trees with arbitrary and variable degree. Then we show that it is possible to implement a distributed version of homomorphisms on binary trees, takingO(n/p+ log2p) parallel time onp < nprocessors, for trees of any skew and taking full account of communication costs. Under slightly stronger restrictions on the underlying functions, this can be improved toO(n/p+ logp). Furthermore, the technique for deriving distributed versions is algebraic, allowing the automatic generation of code for SPMD and data-parallel architectures.

Cilk: An Efficient Multithreaded Runtime System

Cilk (pronounced “silk”) is a C-based runtime system for multithreaded parallel programming. In this paper, we document the efficiency of the Cilk work-stealing scheduler, both empirically and analytically. We show that on real and synthetic applications, the “work” and “critical-path length” of a Cilk computation can be used to model performance accurately. Consequently, a Cilk programmer can focus on reducing the computation's work and critical-path length, insulated from load balancing and other runtime scheduling issues. We also prove that for the class of “fully strict” (well-structured) programs, the Cilk scheduler achieves space, time, and communication bounds all within a constant factor of optimal. The Cilk runtime system currently runs on the Connection Machine CM5 MPP, the Intel Paragon MPP, the Sun Sparcstation SMP, and the Cilk-NOW network of workstations. Applications written in Cilk include protein folding, graphic rendering, backtrack search, and the ★Socrates chess program, which won second prize in the 1995 ICCA World Computer Chess Championship.

An experiment for truly parallel logic programming

A truly parallel logic programming system is proposed. The system is based on the commercially available parallel logic programming language STRAND, which has been extended in order to overcome the inherent limitations of such systems, like AND-type of parallelism, lack of backtracking, limited unification, etc. The system has been tested using an example from the area of natural language processing.

Interfacing Condor and PVM to harness the cycles of workstation clusters

A continuing challenge to the scientific research and engineering communities is how to fully utilize computational hardware. In particular, the proliferation of clusters of high performance workstations has become an increasingly attractive source of compute power. Developments to take advantage of this environment have previously focused primarily on managing the resources, or on providing interfaces so that a number of machines can be used in parallel to solve large problems. Both approaches are desirable, and indeed should be complementary. Unfortunately, the resource management and parallel processing systems are usually developed by independent groups, and they usually do not interact well together. To bridge this gap, we have developed a framework for interfacing these two sorts of systems. Using this framework, we have interfaced PVM, a popular system for parallel programming with Condor, a powerful resource management system. This combined system is operational, and we have made further developments to provide a single coherent environment.

Cuts and side-effects in and-or parallel Prolog

Practical Prolog programs usually contain extra-logical features like cuts, side-effects, and database manipulating predicates. In order to exploit implicit parallelism from real applications while preserving sequential Prolog semantics, a parallel logic programming system should necessarily support these features. In this paper we show how Prolog's extra-logical features can be supported in an and-or parallel logic programming system. We show that to support extra-logical features an and-or parallel logic programming system should recompute the solutions to independent goals instead of sharing them. We describe an abstraction called the composition tree for representing and-or parallel execution with recomputation. We introduce the notion of “focal-leftmostness” in the composition tree and use it for deriving complete and efficient methods for supporting extra-logical predicates in and-or parallel logic programming systems based on the composition tree abstraction.

Parallel computing in networks of workstations with Paralex

Modern distributed systems consisting of powerful workstations and high-speed interconnection networks are an economical alternative to special-purpose supercomputers. The technical issues that need to be addressed in exploiting the parallelism inherent in a distributed system include heterogeneity, high-latency communication, fault tolerance and dynamic load balancing. Current software systems for parallel programming provide little or no automatic support towards these issues and require users to be experts in fault-tolerant distributed computing. The Paralex system is aimed at exploring the extent to which the parallel application programmer can be liberated from the complexities of distributed systems. Paralex is a complete programming environment and makes extensive use of graphics to define, edit, execute, and debug parallel scientific applications. All of the necessary code for distributing the computation across a network and replicating it to achieve fault tolerance and dynamic load balancing is automatically generated by the system. In this paper we give an overview of Paralex and present our experiences with a prototype implementation.

An object-oriented approach to develop software fault-tolerant mechanisms for parallel programming systems

Some parallel programming systems are libraries that allow programmers to write thread-based parallel programs with existing sequential languages. Basically, parallel programs are hard to debug and much more complex than sequential programs which causes design faults to possibly reside in the parallel programs. This paper is aimed to design and implement a software fault-tolerant mechanism in an object-oriented approach for the existing parallel programming systems. With these software fault-tolerant objects, programmers can write their reliable parallel programs on these parallel programming systems. Recover Block, N-Version Programming, and Conversation software fault tolerant mechanisms are chosen to support. All these mechanisms are implemented and grouped into a separate software layer which resides on the top of the parallel programming system, used to monitor the behavior of applications, detect software faults, and recover and restart programs. Parallel programming systems are responsible for managing concurrent threads and for providing fault-tolerant mechanisms with necessary concurrent facilities. This layered system architecture makes these software fault-tolerant mechanisms portable, extensible, and lighter overhead. We have originally implemented the above software fault-tolerant objects based on Presto in C++. These objects have also been ported to C-Thread of Mach and LWP of SUN OS.