Distributed Memory Parallel Machines Research Articles

Exploiting Distributed-Memory and Shared-Memory Parallelism on Clusters of SMPs with Data Parallel Programs

Clusters of SMPs are hybrid-parallel architectures that combine the main concepts of distributed-memory and shared-memory parallel machines. Although SMP clusters are widely used in the high performance computing community, there exists no single programming paradigm that allows exploiting the hierarchical structure of these machines. Most parallel applications deployed on SMP clusters are based on MPI, the standard API for distributed-memory parallel programming, and thus may miss a number of optimization opportunities offered by the shared memory available within SMP nodes. In this paper we present extensions to the data parallel programming language HPF and associated compilation techniques for optimizing HPF programs on clusters of SMPs. The proposed extensions enable programmers to control key aspects of distributed-memory and shared-memory parallelization at a high-level of abstraction. Based on these language extensions, a compiler can adopt a hybrid parallelization strategy which closely reflects the hierarchical structure of SMP clusters by automatically exploiting shared-memory parallelism based on OpenMP within cluster nodes and distributed-memory parallelism utilizing MPI across nodes. We describe the implementation of these features in the VFC compiler and present experimental results which show the effectiveness of these techniques.

An efficient parallel algorithm for O( N2) direct summation method and its variations on distributed-memory parallel machines

We present a novel, highly efficient algorithm to parallelize O( N 2) direct summation method for N-body problems with individual timesteps on distributed-memory parallel machines such as Beowulf clusters. Previously known algorithms, in which all processors have complete copies of the N-body system, has the serious problem that the communication–computation ratio increases as we increase the number of processors, since the communication cost is independent of the number of processors. In the new algorithm, p processors are organized as a p × p two-dimensional array. Each processor has N/ p particles, but the data are distributed in such a way that complete system is presented if we look at any row or column consisting of p processors. In this algorithm, the communication cost scales as N/ p , while the calculation cost scales as N 2/ p. Thus, we can use a much larger number of processors without losing efficiency compared to what was practical with previously known algorithms.

ON MESSAGE PACKAGING IN TASK SCHEDULING FOR DISTRIBUTED MEMORY PARALLEL MACHINES

In this paper, we report a performance gap betweeen a schedule with small makespan on the task scheduling model and the corresponding parallel program on distributed memory parallel machines. The main reason of the gap is the software overhead in the interprocessor communication. Therefore, speedup ratios of schedules on the model do not approximate well to those of parallel programs on the machines. The purpose of the paper is to get a task scheduling algorithm that generates a schedule with good approximation to the corresponding parallel program and with small makespan. For this purpose, we propose algorithm BCSH that generates only bulk synchronous schedules. In those schedules, no-communication phases and communication phases appear alternately. All interprocessor communications are done only in the latter phases, and thus the corresponding parallel programs can make better use of the message packaging technique easily. It reduces many software overheads of messages form a source processor to the same destination processor to almost one software overhead, and improves the performance of a parallel program significantly. Finally, we show some experimental results of performance gaps on BCSH, Kruatrachue's algorithm DSH, and Ahmad et al's algorithm ECPFD. The schedules by DSH and ECPFD are famous for their small makespans, but message packaging can not be effectively applied to the corresponding program. The results show that a bulk synchronous schedule with small makespan has advantages that the gap is small and the corresponding program is a high performance parallel one.

PARSIMONY: An Infrastructure for Parallel Multidimensional Analysis and Data Mining

Multidimensional analysis and online analytical processing (OLAP) operations require summary information on multidimensional data sets. Most common are aggregate operations along one or more dimensions of numerical data values. Simultaneous calculation of multidimensional aggregates are provided by the Data Cube operator, used to calculate and store summary information on a number of dimensions. This is computed only partially if the number of dimensions is large. Query processing for these applications requires different views of data to gain insight and for effective decision support. Queries may either be answered from a materialized cube in the data cube or calculated on the fly. The multidimensionality of the underlying problem can be represented both in relational and in multidimensional databases, the latter being a better fit when query performance is the criteria for judgment. Relational databases are scalable in size for OLAP and multidimensional analysis and efforts are on to make their performance acceptable. On the other hand multidimensional databases have proven to provide good performance for such queries, although they are not very scalable. In this article we address (1) scalability in multidimensional systems for OLAP and multidimensional analysis and (2) integration of data mining with the OLAP framework. We describe our system PARSIMONY, parallel and scalable infrastructure for multidimensional online analytical processing, used for both OLAP and data mining. Sparsity of data sets is handled by using chunks to store data either as a dense block using multidimensional arrays or as sparse representation using a bit encoded sparse structure. Chunks provide a multidimensional index structure for efficient dimension oriented data accesses much the same as multidimensional arrays do. Operations within chunks and between chunks are a combination of relational and multidimensional operations depending on whether the chunk is sparse or dense. Further, we develop parallel algorithms for data mining on the multidimensional cube structure for attribute-oriented association rules and decision-tree-based classification. These take advantage of the data organization provided by the multidimensional data model. Performance results for high dimensional data sets on a distributed memory parallel machine (IBM SP-2) show good speedup and scalability.

Efficient Support for Pipelining in Software Distributed Shared Memory Systems

Though more difficult to program, distributed-memory parallel machines provide greater scalability than their shared-memory counterparts. Software Distributed Shared Memory (SDSM) systems provide the abstraction of shared memory on a distributed machine. While SDSMs provide an attractive programming model, they currently cannot efficiently support all classes of scientific application. One such class are those with recurrences that cause dependencies across processors or nodes. A popular solution to such problems is to use pipelining which breaks the computation into blocks; each processor performs the computation of a block, which enables the next processor in the pipeline to compute its corresponding block. Once the pipeline is filled, the computation of blocks proceeds in parallel. While pipelining is useful, it is not efficiently supported by current SDSM systems. This paper presents an approach to integrating pipelining into SDSM systems. We describe our design and implementation of one-way pipelining in a SDSM. The key idea is to retain the shared-memory model, but design the extensions such that the execution will mimic what would be done in an explicit message-passing program. We show that one-way pipelining is superior to the two most common ways to program pipelined applications, which are distributed locks and explicit matrix transposition. Finally, we show that one-way pipelining is competitive with a hand-coded, explicit message-passing program.

Visualization of large data sets with the Active Data Repository

We implement ray-casting-based volume rendering and isosurface rendering methods using the Active Data Repository (ADR) for visualizing out-of-core data sets. We have developed the ADR object-oriented framework to provide support for applications that employ range queries with user-defined mapping and aggregation operations on large-scale multidimensional data. ADR targets distributed-memory parallel machines with one or more disks attached to each node. It is designed as a set of modular services implemented in C++, which can be customized for application-specific processing. The ADR runtime system supports common operations such as memory management, data retrieval, and scheduling of processing across a parallel machine.

On designing optimal parallel triangular solvers

This paper explores the problem of solving triangular linear systems on parallel distributed-memory machines. Working within the LogP model, tight asymptotic bounds for solving these systems using forward/backward substitution are presented. Specifically, lower bounds on execution time independent of the data layout, lower bounds for data layouts in which the number of data items per processor is bounded, and lower bounds for specific data layouts commonly used in designing parallel algorithms for this problem are presented in this paper. Furthermore, algorithms are provided which have running times within a constant factor of the lower bounds described. One interesting result is that the popular two-dimensional block matrix layout necessarily results in significantly longer running times than simpler one-dimensional schemes. Finally, a generalization of the lower bounds to banded triangular linear systems is presented.

A direct parallel implementation of the Hoshen–Kopelman algorithm for distributed memory architectures

We present a direct (i.e. non iterative) method for identifying the clusters of a cubic lattice on a distributed memory parallel machine. We describe the various cluster properties which we were able to compute with this method. Finally, we give some performance measurements.

Automatic mapping of system of N-dimensional affine recurrence equations (SARE) onto distributed memory parallel systems

The automatic extraction of parallelism from algorithms, and the consequent parallel code generation, is a challenging problem. We present a procedure for automatic parallel code generation in the case of algorithms described through a SARE (Set of Affine Recurrence Equations). Starting from the original SARE description in an N-dimensional iteration space, the algorithm is converted into a parallel code for an (eventually virtual) m-dimensional distributed memory parallel machine (m<N). We demonstrate some theorems which are the mathematical basis for the proposed parallel generation tool. The projection technique used in the tool is based on the polytope model. Some affine transformations are introduced to project the polytope from the original iteration space onto another polytope, preserving the SARE semantic, in the time-processor (t,p) space. Points in (t,p) are individuated through the m-dimensional p coordinate and the n-dimensional t coordinate, resulting in N=n+m. Along with polytope transformation, a methodology to generate the code within processors is given. Finally, a cost function, used to guide the heuristic search for the polytope transformation and derived from the actual implementation of the method on an MPP SIMD machine, is introduced.

High-Performance Computing for Large-Scale Analysis, Optimization, and Control

Research papers published on parallel processing, supercomputing, and distributed computing since 1994 are reviewed in this article. The focus is on review of the archival journal articles published in three areas: analysis, optimization, and control. The review is divided into three main sections: (1) parallel processing on dedicated shared memory and distributed memory parallel machines; (2) distributed computing on a cluster of networked workstations; and (3) parallel computing and object-oriented programming. The great majority of the journal articles published since 1987 deal with fundamental issues and academic problems. The trend in the future, at least in part, should be more toward solution of large-scale and complicated real-life engineering problems, the kind of problems that cannot be solved readily by conventional uniprocessor computers.

Tensorial Basis Spline Collocation Method for Poisson's Equation

This paper aims to describe the tensorial basis spline collocation method applied to Poisson's equation. In the case of a localized 3D charge distribution in vacuum, this direct method based on a tensorial decomposition of the differential operator is shown to be competitive with both iterative BSCM and FFT-based methods. We emphasize the O(h4) and O(h6) convergence of TBSCM for cubic and quintic splines, respectively. We describe the implementation of this method on a distributed memory parallel machine. Performance measurements on a Cray T3E are reported. Our code exhibits high performance and good scalability: As an example, a 27 Gflops performance is obtained when solving Poisson's equation on a 2563 non-uniform 3D Cartesian mesh by using 128 T3E-750 processors. This represents 215 Mflops per processors.

An Interleaving Transformation for Parallelizing Reductions for Distributed-Memory Parallel Machines

Reduction operations frequently appear in algorithms. Due to their mathematical invariance properties (assuming that round-off errorscan be tolerated), it is reasonable to ignore ordering constraints on the computation of reductions in order to take advantage of the computing power of parallel machines. One obvious and widely-used compilation approach for reductions is syntactic pattern recognition. Either the source language includes explicit reduction operators, or certain specific loops are recognized as equivalent to known reductions. Once such patterns are recognized, hand optimized code for the reductions are incorporated in the target program. The advantage of this approach is simplicity. However, it imposes restrictions on the reduction loops—no data dependence other than that caused by the reduction operation itself is allowed in the reduction loops. In this paper, we present a parallelizing technique, interleaving transformation, for distributed-memory parallel machines. This optimization exploits parallelism embodied in reduction loops through combination of data dependence analysis and region analysis. Data dependence analysis identifies the loop structures and the conditions that can trigger this optimization. Region analysis divides the iteration domain into a sequential region and an order-insensitive region. Parallelism is achieved by distributing the iterations in the order-insensitive region among multiple processors. We use a triangular solver as an example to illustrate the optimization. Experimental results on various distributed-memory parallel machines, including the Connection Machines CM-5, the nCUBE, the IBM SP-2, and a network of Sun Workstations are reported.

Parallel solution of three-dimensional Marangoni flow in liquid bridges

SUMMARY This paper describes the implementation and performances of a parallel solver for the direct numerical simulation of the three-dimensional and time-dependent Navier‐Stokes equations on distributedmemory, massively parallel computers. The feasibility of this approach to study Marangoni flow instability in half zone liquid bridges is examined. The results indicate that the incompressible, non-linear Navier‐Stokes problem, governing the Marangoni flows behavior, can effectively be parallelized on a distributed memory parallel machine by remapping the distributed data structure. The numerical code is based on a three-dimensional Simplified Marker and Cell (SMAC) primitive variable method applied to a staggered finite difference grid. Using this method, the problem is split into two problems, one parabolic and the other elliptic A parallel algorithm, explicit in time, is utilized to solve the parabolic equations. A parallel multisplitting kernel is introduced for the solution of the pseudo pressure elliptic equation, representing the most time-consuming part of the algorithm. A grid-partition strategy is used in the parallel implementations of both the parabolic equations and the multisplitting elliptic kernel. A Message Passing Interface (MPI) is coded for the boundary conditions; this protocol is portable to different systems supporting this interface for interprocessor communications. Numerical experiments illustrate good numerical properties and parallel efficiency. In particular, good scalability on a large number of processors can be achieved as long as the granularity of the parallel application is not too small. However, increasing the number of processors, the Speed-Up is ever smaller than the ideal linear Speed-Up. The communication timings indicate that complex practical calculations, such as the solutions of the Navier‐Stokes equations for the numerical simulation of the instability of Marangoni flows, can be expected to run on a massively parallel machine with good efficiency. Copyright © 1999 John Wiley & Sons, Ltd.

Finite element analysis of fibre reinforced polymer shells using higher-order shear deformation theories on parallel distributed memory machines

The analysis of fibre reinforced polymer composite (FRPC) structures requires special analytical formulation. An approach based on a higher-order shear deformation theory has been developed and used. Generally this analysis, for a real life problem, imposes great demands on computational requirements. In the present work, the emphasis is on the implementation of FRP analysis software on parallel distributed-memory machine. The parallelisation strategy adopted is explained and some case studies are presented. The performance with respect to number of processors is given and the effectiveness of the parallelisation for solving larger problems is demonstrated.

Large-scale parallel geophysical algorithms in Java: a feasibility study

Java is often accused of being too slow for serious programming, especially for scientific problem solving. However, we found that for a large-scale geophysical application, Java code compiled with current just-in-time compilers runs slower than Fortran by a factor of at most 4, on both a shared-memory parallel machine (SGI Origin2000) and a distributed-memory parallel machine (IBM SP/2). The moderate slow-down is easily offset by the following advantages: (a) object-oriented Java code is easier to maintain and reuse than Fortran code; (b) Java code is fully portable, even among parallel computers with different memory models. Furthermore, better compiler technology is on the horizon, which will narrow the performance gap even more. © 1998 John Wiley & Sons, Ltd.

Large-scale parallel geophysical algorithms in Java: a feasibility study

Java is often accused of being too slow for serious programming, especially for scientific problem solving. However, we found that for a large-scale geophysical application, Java code compiled with current just-in-time compilers runs slower than Fortran by a factor of at most 4, on both a shared-memory parallel machine (SGI Origin2000) and a distributed-memory parallel machine (IBM SP/2). The moderate slow-down is easily offset by the following advantages: (a) object-oriented Java code is easier to maintain and reuse than Fortran code; (b) Java code is fully portable, even among parallel computers with different memory models. Furthermore, better compiler technology is on the horizon, which will narrow the performance gap even more. © 1998 John Wiley & Sons, Ltd.

Virtual signal processors

The introduction of SRAM-based field programmable gate arrays (FPGAs) has opened up a new dimension to parallel computing architectures. This paper describes an alternative approach to parallel computing — reconfigurable or virtual parallel processing (VPP). Rather than mapping an application onto a given parallel machine, the WP approach synthesizes the appropriate type and number of processing elements, as well as the interconnection topology, that is optimal for the application. For each application, configuration data is downloaded to the machine that personalizes the hardware for the task at hand. The paper provides a brief description of the authors reconfigurable computer, Archimedes. The benefits of the VPP approach are highlighted by two example applications — the 2-D FFT and a narrow-band digital filter. A novel parallel implementation of a polynomial transform based 2-D transform is described and compared with results for distributed memory parallel machines that have been reported in the literature. The comparison highlights the computational advantage provided by reconfigurable computing. The digital filter implementation employs sigma–delta modulation encoding to reduce the arithmetic workload. The application of this technology to a communications receiver is explained.

Interprocedural partial redundancy elimination with application to distributed memory compilation

Partial Redundancy Elimination (PRE) is a general scheme for suppressing partial redundancies which encompasses traditional optimizations like loop invariant code motion and redundant code elimination. In this paper, we address the problem of performing this optimization interprocedurally. We present an Interprocedural Partial Redundancy Elimination (IPRE) scheme based upon a new, concise, full program representation. Our method is applicable to arbitrary recursive programs. We use interprocedural partial redundancy elimination for placement of communication and communication preprocessing statements while compiling for distributed memory parallel machines. We have implemented our scheme as an extension to the Fortran D compilation system. We present experimental results from two codes compiled using our system to demonstrate the useful of IPRE in distributed memory compilation.

A PVM-based distributed parallel symbolic system

The purpose of this work is to provide a parallel programming environment for symbolic applications suited to run on Distributed Memory Parallel Systems (DMPS). This paper describes the implementation of such an environment that is based on Sabot's Paralation model (in particular its LISP implementation, Paralation LISP). In our opinion, the clear distinction between the communication and computation aspects in the Paralation model allows a clearer and easier exploitation of the opportunities of parallelism in symbolic applications. Furthermore, the simple semantics of its parallel constructs seem well suited to run on a community of loosely coupled LISP interpreters. We chose to implement the Paralation model constructs on the top of the Parallel Virtual Machine (PVM) package because this tool allows one to have a cheap virtual distributed memory parallel machine simply by using some LAN interconnected powerful workstations.

Compiling dynamic mappings with array copies

Array remapping are useful to many applications on distributed memory parallel machines. They are available in High Performance Fortran, a Fortran-based data-parallel language. This paper describes techniques to handle dynamic mappings through simple array copies: array remapping are translated into copies between statically mapped distinct versions of the array. It discusses the language restrictions required to do so. The remapping graph which captures all remapping and liveness information is presented, as well as additional data-flow optimizations that can be performed on this graph, so as to avoid useless remapping at run time. Such useless remapping appear for arrays that are not used after a remapping. Live array copies are also kept to avoid other flow-dependent useless remappings. Finally the code generation and runtime required by our scheme are discussed. These techniques are implemented in our prototype HPF compiler.