Data-parallel Programming Research Articles

OPUS: HETEROGENEOUS COMPUTING WITH DATA PARALLEL TASKS

The coordination language Opus is an object-based extension of High Performance Fortran (HPF) that supports the integration of coarse-grain task parallelism with HPF-style data parallelism. In this paper we discuss Opus in the Context of multidisciplinary applications (MDAs) which execute in a heterogencous environment. After outlining the major properties of such applications and a number of different approaches towards providing language and tool support for MDAs we describe the salientfeatures of Opus and its implementation, emphasizing the issues related to the coordination of data-parallel HPF programs in a heterogencous environment.

Integrated range comparison for data-parallel compilation systems

A major difficulty in restructuring compilation, and in parallel programming in general, is how to compare parallel performance over a range of system and problem sizes. Execution time varies with system and problem size and an initially fast implementation may become slow when system and problem size scale up. This paper introduces the concept of range comparison. Unlike conventional execution time comparison in which performance is compared for a particular system and problem size, range comparison compares the performance of programs over a range of ensemble and problem sizes via scalability and performance crossing point analysis. A novel algorithm is developed to predict the crossing point automatically. The correctness of the algorithm is proven and a methodology is developed to integrate range comparison into restructuring compilations for data-parallel programming. A preliminary prototype of the methodology is implemented and tested under Vienna Fortran Compilation System. Experimental results demonstrate that range comparison is feasible and effective. It is an important asset for program evaluation, restructuring compilation, and parallel programming.

Buffer-Safe and Cost-Driven Communication Optimization

This paper presents a new approach for optimizing communication of data parallel programs. Our techniques are based on unidirectional bit-vector data flow analyses that enable vectorizing, coalescing and aggregating communication, and overlapping communication with computation both within and across loop nests. Previous techniques are based on fixed communication optimization strategies whose quality is very sensitive to changes of machine and problem sizes. Our algorithm is novel in that we carefully examine tradeoffs between enhancing communication latency hiding and reducing the number and volume of messages by systematically evaluating a reasonable set of promising communication placements for a given program covering several (possibly conflicting) communication guiding profit motives. We useP3T, a state-of-the-art performance estimator, to ensure communication buffer safety and to find the best communication placement of all created ones. First results show that our method implies a significant reduction in communication costs and demonstrate the effectiveness of this analysis in improving the performance of programs.

Data-parallel tomographic reconstruction: A comparison of filtered backprojection and direct Fourier reconstruction

We consider the parallelization of two standard 2D reconstruction algorithms, filtered backprojection and direct Fourier reconstruction, using the data-parallel programming style. The algorithms are implemented on a Connection Machine CM-5 with 16 processors and a peak performance of 2 Gflop/s.

A task- and data-parallel programming language based on shared objects

Many programming languages support either task parallelism, but few languages provide a uniform framework for writing applications that need both types of parallelism or data parallelism. We present a programming language and system that integrates task and data parallelism using shared objects. Shared objects may be stored on one processor or may be replicated. Objects may also be partitioned and distributed on several processors.Task parallelism is achieved by forking processes remotely and have them communicate and synchronize through objects. Data parallelism is achieved by executing operations on partitioned objects in parallel. Writing task-and data-parallel applications with shared objects has several advantages. Programmers use the objects as if they were stored in a memory common to all processors. On distributed-memory machines, if objects are remote, replicated, or partitioned, the system takes care of many low-level details such as data transfers and consistency semantics. In this article, we show how to write task-and data-parallel programs with our shared object model. We also desribe a portable implementation of the model. To assess the performance of the system, we wrote several applications that use task and data parallelism and excuted them on a collection of Pentium Pros connected by Myrinet. The performance of these applications is also discussed in this article.

Compilation techniques for out-of-core parallel computations

The difficulty of handling out-of-core data limits the performance of supercomputers as well as the potential of the parallel machines. Since writing an efficient out-of-core version of a program is a difficult task and virtual memory systems do not perform well on scientific computations, we believe that there is a clear need for compiler directed explicit I/O approach for out-of-core computations. In this paper, we first present an out-of-core compilation strategy based on a disk storage abstraction. Then, we offer a compiler algorithm to optimize locality of disk accesses in out-of-core codes by choosing a good combination of file layouts on disks and loop transformations. We introduce memory coefficient and processor coefficient concepts to characterize the behavior of out-of-core programs under different memory constraints. We also enhance our algorithm to handle data-parallel programs which contain multiple loop nest. Our initial experimental results obtained on IBM SP-2 and Intel Paragon provide encouraging evidence that our approach is successful at optimizing programs which depend on disk-resident data in distributed-memory machines.

Using integer sets for data-parallel program analysis and optimization

In this paper, we describe our experience with using an abstract integer-set framework to develop the Rice dHPF compiler, a compiler for High Performance Fortran. We present simple, yet general formulations of the major computation partitioning and communication analysis tasks as well as a number of important optimizations in terms of abstract operations on sets of integer tuples. This approach has made it possible to implement a comprehensive collection of advanced optimizations in dHPF, and to do so in the context of a more general computation partitioning model than previous compilers. One potential limitation of the approach is that the underlying class of integer set problems is fundamentally unable to represent HPF data distributions on a symbolic number of processors. We describe how we extend the approach to compile codes for a symbolic number of processors, without requiring any changes to the set formulations for the above optimizations. We show experimentally that the set representation is not a dominant factor in compile times on both small and large codes. Finally, we present preliminary performance measurements to show that the generated code achieves good speedups for a few benchmarks. Overall, we believe we are the first to demonstrate by implementation experience that it is practical to build a compiler for HPF using a general and powerful integer-set framework.

Parallel Processing And Fingerprint Image Comparison

A neural-network-based fingerprint image comparison (FIC) system is presented, and a variety of approaches for parallel simulation of the FIC system are explored. The FIC system contains an image preprocessing stage and a neural network decision stage. The preliminary training and testing results of the FIC system using the NIST fingerprint database lire discussed. The parallel simulation algorithms of the neural network presented in this paper include data parallel programming (SIMD) and message-passing programming (MIMD) implemented on the CM-5. The results of the performance analysis show the advantages of the MIMD approach: it is effective, flexible, and can avoid detailed data distribution mechanisms. On the other hand, the SIMD implementation presents worse performance than sequential because of the existence of extensive data routing overhead.

The NRL layered ocean model

We describe the portable scalable implementation of the NRL Layered Ocean Model (NLOM). Scalability is based primarily on the tiled data parallel parallel programming paradigm. This is sufficiently general that the actual technique used on a given machine to obtain scalability can be selected at compile time from: (i) data parallel, (ii) SPMD message passing, (iii) autotasking, or (iv) SPMD message passing between multi-processor autotasked systems. The code is thus portable onto all machine types likely to be used by ocean modelers.

High Performance Fortran, Version 2

This paper introduces the ideas that underly the data-parallel language High Performance Fortran (HPF) and the new ideas in version 2 of HPF. It first reviews HPF's key language elements. It discusses the meaning of data parallelism and the limitations of HPF version 1 as a data-parallel programming language. The second part of the paper is a review of the development of version 2 of HPF. The extended language, under development in 1996, includes a richer data mapping capability; an extension to the independent loop that allows reduction operations in the loop range; a means for directing the mapping of computation as well as data; and a way to specify concurrent execution of several parallel tasks on disjoint subsets of processors.

An integrated performance analysis tool for SPMD data-parallel programs

A new generation of data parallel languages have been proposed whereby a user specifies how data structures are to be distributed amongst the processor nodes of a distributed-memory machine. Based on this information, the compiler then generates code for the parallel application. Although this approach significantly simplifies the development of the initial version of a parallel application, selection of good data distributions leading to efficient computations is often quite difficult. Therefore, performance debuggers are needed to yield insights into the data distribution effects. On the other hand, most of the existing approaches to performance debugging are very general and thus do not provide the user feedback in terms of the high level programming model or the source code of the parallel application. In this paper, we describe a novel approach to performance debugging of data parallel programs. The design and implementation of a visual performance debugger is described that is specifically targeted to meet the performance debugging requirements of a data-parallel programming model based on user-specified data distributions. The performance debugger is part of an integrated programming environment, called EPPP, which also supports a data parallel compiler and a parallel architecture simulator. Thus development and performance debugging of an application may be done either on the real hardware or by using the simulator. The code for EPPP can be obtained free of cost (contact web address, http://www.crim.ca/apar).

Scalable parallel Benders decomposition for stochastic linear programming

We develop a scalable parallel implementation of the classical Benders decomposition algorithm for two-stage stochastic linear programs. Using a primal-dual, path-following algorithm for solving the scenario subproblems we develop a parallel implementation that alleviates the difficulties of load balancing. Furthermore, the dual and primal step calculations can be implemented using a data-parallel programming paradigm. With this approach the code effectively utilizes both the multiple, independent processors and the vector units of the target architecture, the Connection Machine CM-5. The, usually limiting, master program is solved very efficiently using the interior point code LoQo on the front-end workstation. The implementation scales almost perfectly with problem and machine size. Extensive computational testing is reported with several large problems with up to 2 million constraints and 13.8 million variables.

Determining the Execution Time Distribution for a Data Parallel Program in a Heterogeneous Computing Environment

An important problem in heterogeneous computing (HC) is predicting task execution time. A methodology is introduced for determining the execution time distribution for a given data parallel program that is to be executed in an SIMD, MIMD (SPMD), and/or mixed-mode SIMD/MIMD (SPMD) HC environment. The program is assumed to contain operations and constructs whose execution times depend on input-data values. The methodology uses a block-based approach to transform the program into a flow analysis tree and computes the execution time distribution for the program, given the execution modes for each node in the flow analysis tree, an estimated execution time distribution for each operation in both modes, and appropriate probabilistic models for control and data conditional constructs. The results are directly applicable to both mixed-machine and mixed-mode HC systems.

A new model for integrated nested task and data parallel programming

High Performance Fortran (HPF) has emerged as a standard language fordata parallel computing. However, a wide variety of scientific applications are best programmed by a combination of task and data parallelism. Therefore, a good model of task parallelism is important for continued success of HPF for parallel programming. This paper presents a task parallelism model that is simple, elegant, and relatively easy to implement in an HPF environment. Task parallelism is exploited by mechanisms for dividing processors into subgroups and mapping computations and data onto processor subgroups. This model of task parallelism has been implemented in the Fx compiler at Carnegie Mellon University. The paper addresses the main issues in compiling integrated task and data parallel programs and reports on the use of this model for programming various flat and nested task structures. Performance results are presented for a set of programs spanning signal processing, image processing, computer vision and environment modeling. A variant of this task model is a new approved extension of HPF and this paper offers insight into the power of expression and ease of implementation of this extension.

Experiments with ‘HP Java’

We consider the possible role of Java as a language for high performance computing. After discussing reasons why Java may be a natural candidate for a portable parallel programming language, we describe several case studies. These cover Java socket programming, message-passing through a Java interface to MPI, and class libraries for data-parallel programming in Java. © 1997 John Wiley & Sons, Ltd.

Static Analysis to Reduce Synchronization Costs Data-Parallel Programs with Remote Memory Copy

For a program with sufficient parallelism, reducing synchronization costs is an important objective for achieving efficient execution. This paper presents a novel methodology for reducing synchronization costs of programs compiled for SPMD execution. This methodology combines data flow analyisis with communication analysis to determine the ordering between production and consumption of data on different processors, which helps in identifying redundant synchronization. The resulting framework is more powerful than any that have been previously presented, as it provides the first algorithm that can eliminate synchronization messages even from computations that need communication. We show that several commonly occuring computation patterns such as reductions and stencil computations with reciprocal producer-consumer relationship between processors lend themselves well to this optimization, an observation that is confirmed by an examination of some HPE benchmark programs. Our framework also recognizes situations where the synchronization needs for multiple data transfers can be satisfied by a single synchronization message. This analysis, while applicable to all shared memory machines as well, is especially useful for those with a flexible cache-coherence protocol, as it identifies efficient ways of moving data directly from producers to consumers, often without any extra synchronization.

Experiments with ‘HP Java’

We consider the possible role of Java as a language for high performance computing. After discussing reasons why Java may be a natural candidate for a portable parallel programming language, we describe several case studies. These cover Java socket programming, message-passing through a Java interface to MPI, and class libraries for data-parallel programming in Java. © 1997 John Wiley & Sons, Ltd.

Dynamic data decomposition in a message-passing environment

Abstract The performance of a data parallel program is critically dependent on the data decomposition that the programmer chooses at implementation time. This choice must take into account a combination of different factors such as the kind of the problem, the machine architecture and the data domain size. When these elements are known before execution, the programmer can adopt traditional message-passing languages and optimise performance by means of programs which are dependent on the chosen data decomposition. On the other hand, when the factors that determine the best decomposition are known at run-time only, adequate efficiency can be achieved by a code that dynamically adapts its computation/communication pattern to various decompositions. To assist the programmer in the implementation of a decomposition-independent code, we propose a new programming environment, namely PLUS. It provides the programmer with message-passing primitives that avoid the specification of a data decomposition during the implementation phase, and a run-time support that permits dynamic changes among regular decompositions without affecting the implemented program.

Which approach to parallelizing scientific codes — That is the question

We present in this paper the strong points and limitations of semi-automatic parallelization, data parallel programming and message passing programming. We apply these on two numerical algorithms namely a bi-dimensional Fourier transform algorithm and a conjugate gradient programs. We implemented this program for each of the different methods on a Cray T3D. The results of these experiments demonstrate the accuracy of our proposition that when the three methods are combined, efficiency, portability and easiness of parallel programming may be achieved.

Load Balancing and Performance Issues for Data Parallel

A data parallel program is presented that solves the reactive Euler equations for stiff chemical nonequilibrium ows on Connection Machines CM-2/200 and CM-5/5E. The program is written in CM Fortran and uses direction and time-step splitting to couple representations of the chemical and uid dynamicprocesses on a structured Cartesian grid.An explicit high-ordermonotonealgorithmwith nonlineardamping is used to integrate the convection terms, and a hybrid asymptotic/modi® ed-Euler approach is used to solve the system of ordinary differential equations from the chemical source terms. Integration of the uid dynamics was conservatively determined to be 9.4 and 12.0 G ops on a 512-node CM-5 and CM-5E, respectively. The uid dynamics solver scaled well for large problems; however, both the performance and the scaling are signi® cantly affected by the nearest-neighbor communications, which accounted for at least 24% of the execution time on the CM-5. For the integration of the chemistry source terms, poor load balancing signi® cantly affected performance of the program. Therefore, a new load-balancing algorithmwas developed that reduces the chemistry integration time by a factor of six for the test problem, a detonation propagating in a hydrogen± oxygen± argon mixture. Moreover, the chemistry integration time, with the load balancing, is slightly less than the time required to integrate the uid dynamics. As a result, an ef® cient data parallel program for solving stiff chemical nonequilibrium ows is available for problems that were too expensive to solve in the past.