Distributed Memory Multicomputers Research Articles

Synthesis and optimization of a nonideal distillation system using a parallel genetic algorithm

The automated synthesis of separation sequences for nonideal mixtures has only recently been attempted. The need for rigorous physical property estimation procedures along with the possibly complex recycle structures leads to the specification of a nonlinear programming problem, one in which both the objective function and the feasibility region are nonconvex. This paper describes the use of a stochastic optimization procedure, genetic algorithms, for the optimization of a preselected sequence of distillation units for the separation of a three component azeotropic mixture. By fixing the structure in terms of the units desired, the optimization problem is reduced to one of designing each of the relevant units, choosing the appropriate operating conditions, and finding the optimal heat exchange network. To tackle the high computational effort required, the implementation uses a distributed memory multicomputer in the form of a network of workstations.

Techniques for compiling programs on distributed memory multicomputers

It is widely accepted that distributed memory parallel computers will play an important role in solving computation-intensive problems. However, the design of an algorithm in a distributed memory system is time-consuming and error-prone, because a programmer is forced to manage both parallelism and communication. In this paper, we present techniques for compiling programs on distributed memory parallel computers. We will study the storage management of data arrays and the execution schedule arrangement of Do-loop programs on distributed memory parallel computers. First, we introduce formulas for representing data distribution of specific data arrays across processors. Then, we define communication cost for some message-passing communication operations. Next, we derive a dynamic programming algorithm for data distribution. After that, we show how to improve the communication time by pipelining data, and illustrate how to use data-dependence information for pipelining data. Jacobi's iterative algorithm and the Gauss elimination algorithm for linear systems are used to illustrate our method. We also present experimental results on a 32-node nCUBE-2 computer.

Parallelization of a Monte Carlo algorithm for the simulation of polymer melts

The Continuum-configurational Bias Monte Carlo algorithm (CBMC) is a very efficient method for the simulation of polymer melts. This algorithm is well-suited to parallelization on shared-memory multiprocessors. The main effort for parallelization is often spent in the decomposition of the data structures and the design of the parallel program sections. We present some generally applicable methods to improve the optimization process on shared-memory multiprocessors using global address space and on distributed-memory multicomputers using message passing.

Design and performance of a scalable parallel community climate model

We describe the design of a parallel global atmospheric circulation model, PCCM2. This parallel model is functionally equivalent to the National Center for Atmospheric Research's Community Climate Model, CCM2, but is structured to exploit distributed memory multi-computers. PCCM2 incorporates parallel spectral transform, semi-Lagrangian transport, and load balancing algorithms. We present detailed performance results on the IBM SP2 and Intel Paragon. These results provide insights into the scalability of the individual parallel algorithms and of the parallel model as a whole.

The Paradigm compiler for distributed-memory multicomputers

To harness the computational power of massively parallel distributed-memory multicomputers, users must write efficient software. This process is laborious because of the absence of global address space. The programmer must manually distribute computations and data across processors and explicitly manage communication. The Paradigm (PARAllelizing compiler for DIstributed-memory, General-purpose Multicomputers) project at the University of Illinois addresses this problem by developing automatic methods for the efficient parallelization of sequential programs. A unified approach efficiently supports regular and irregular computations using data and functional parallelism.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Optimization of array redistribution for distributed memory multicomputers

Languages such as High Performance Fortran implement parallel algorithms by distributing large data structures across a multicomputer system. To enhance parallelism and reduce communication, it is sometimes beneficial for a programmer to change the distribution between phases of the algorithm. We introduce a new mapping strategy, called the spiral mapping, that reduces the communication overhead of array redistribution. Redistribution using the spiral mapping exploits communication locality and reduces global communication conflicts. We implemented redistribution using the standard linear mapping and the spiral mapping for two dimensional arrays; for 1024 × 1024 arrays, redistribution using the spiral mapping is 36% faster than using the linear mapping on a 16 node Intel iPSC/860.

Portable code for process synthesis using workstation clusters and distributed memory multicomputers

The automated synthesis of process flowsheets is a computationally intensive task. CHiPS is a process synthesis package which takes account of heat integration and stream states at the synthesis stage. By discretizing all allowable flows and heat exchanges, a dynamic programming and branch and bound approach can be used to find the optimal flowsheet. In this paper, we describe a parallel implementation of CHiPS that uses the dynamic programming technique only. Good speedups are obtained relative to the best method on serial machines. The implementation is portable across a range of architectures, including clusters of Unix workstations. We show that the communication and load balancing properties of the program make it suitable for workstation clusters.

Partitioning and mapping of nested loops for linear array multicomputers

In distributed-memory multicomputers, minimizing interprocessor communication is the key to the efficient execution of parallel programs. In order to reduce the amount of communication overhead, parallel programs on multicomputers must be carefully scheduled by parallelizing compilers. This paper proposes some compilation techniques for partitioning and mapping nested loops with constant data dependences onto linear array multicomputers. First, a systematic partition strategy is proposed to project ann-dimensional computational structure, representing ann-nested loop, onto a line to form a one-dimensional projected structure with low communication overhead. Then, a mapping algorithm is proposed for mapping the partitioned loops onto linear arrays in a way that balances the workload and minimizes the communication cost among processors. Finally, parallel execution codes can be automatically generated for such linear array multicomputers.

Automatic partitioning of parallel loops and data arrays for distributed shared-memory multiprocessors

Presents a theoretical framework for automatically partitioning parallel loops to minimize cache coherency traffic on shared-memory multiprocessors. While several previous papers have looked at hyperplane partitioning of iteration spaces to reduce communication traffic, the problem of deriving the optimal tiling parameters for minimal communication in loops with general affine index expressions has remained open. Our paper solves this open problem by presenting a method for deriving an optimal hyperparallelepiped tiling of iteration spaces for minimal communication in multiprocessors with caches. We show that the same theoretical framework can also be used to determine optimal tiling parameters for both data and loop partitioning in distributed memory multicomputers. Our framework uses matrices to represent iteration and data space mappings and the notion of uniformly intersecting references to capture temporal locality in array references. We introduce the notion of data footprints to estimate the communication traffic between processors and use linear algebraic methods and lattice theory to compute precisely the size of data footprints. We have implemented this framework in a compiler for Alewife, a distributed shared-memory multiprocessor.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Relaxing synchronization in distributed simulated annealing

This paper presents a cost error measurement scheme and relaxed synchronization method, for simulated annealing on a distributed memory multicomputer, which predicts the amount of cost error that an algorithm will tolerate. An adaptive error control method is developed and implemented on an Intel iPSC/2.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Pipeline architectures for recursive morphological operations

Introduces efficient pipeline architectures for the recursive morphological operations. The standard morphological operation is applied directly on the original input image and produces an output image. The order of image scanning in which the operator is applied to the input pixels is irrelevant. However, the intent of the recursive morphological operations is to feed back the output at the current scanning pixel to overwrite its corresponding input pixel to be considered into computation at the following scanning pixels. The resultant output image by recursive morphology inherently depends on the image scanning sequence. Two pipelined implementations of the recursive morphological operations are presented. The design of an application-specific systolic array is first introduced. The systolic array uses 3 n cells to process an nxn image in 6 n-2 cycles. The cell utilization rate is 100%. Second, a parallel program implementing the recursive morphological operations and running on distributed-memory multicomputers is described. Performance of the program can be finely tuned by choosing appropriate partition parameters.

Genetics-based learning of new heuristics: rational scheduling of experiments and generalization

We present new methods for the automated learning of heuristics in knowledge lean applications and for finding heuristics that can be generalized to unlearned domains. These applications lack domain knowledge for credit assignment; hence, operators for composing new heuristics are generally model free, domain independent, and syntactic in nature. The operators we have used are genetics based; examples of which include mutation and cross over. Learning is based on a generate and test paradigm that maintains a pool of competing heuristics, tests them to a limited extent, creates new ones from those that perform well in the past, and prunes poor ones from the pool. We have studied three important issues in learning better heuristics: anomalies in performance evaluation; rational scheduling of limited computational resources in testing candidate heuristics in single objective as well as multiobjective learning; and finding heuristics that can be generalized to unlearned domains. We show experimental results in learning better heuristics for: process placement for distributed memory multicomputers, node decomposition in a branch and bound search, generation of test patterns in VLSI circuit testing, and VLSI cell placement and routing. >

Dynamo ‐ a portable tool for dynamic load balancing on distributed memory multicomputers

AbstractDynamic load balancing is an important technique when developing applications with unpredictable load distribution on distributed memory multicomputers. A tool, Dynamo, that can be used to utilize dynamic load balancing is presented. This tool separates the application from the load balancer and thus makes it possible to easily exchange the load balancer of a given application and experiment with different load balancing strategies. A prototype of Dynamo has been implemented in the C language on an Intel iPSC/2 Hypercube. Dynamo is demonstrated by two example programs. The first program solves the N queen problem using a backtracking algorithm and the second solves a 0‐1 knapsack problem using a depth‐first branch and bound algorithm.

GLOBAL SYNCHRONIZATION ALGORITHMS FOR THE INTEL IPSC/860

In a distributed memory multicomputer that has no global clock, global processor synchronization can only be achieved through software. Global synchronization is used in many applications, including tridiagonal systems solvers, CFD codes, and sequence comparison algorithms. For the Intel iPSC/860 in particular, global synchronization can also be used to ensure the most effective use of the communication network. Two global synchronization algorithms are considered for the iPSC/860: The gsync primitive provided by Intel and the RDS algorithm. Based on the communication model presented here, it is shown that gsync sometimes leaves the processors more poorly synchronized than they were to begin with. It is also shown that interrupts from the node operating system can cause gsync to contend for communication ports with the application code. The RDS algorithm does not have these shortcomings and costs only slightly more than the other algorithms. Measurements of the cost of message shift operations preceded by global synchronization confirm that the RDS algorithm always synchronizes the nodes more precisely than gsync.

Performance of the node/link behaviour of multicomputer networks with input buffer limiting and channel contention

Efficient interprocessor communication is a key parameter to achieve high performance in distributed memory multicomputer networks. Thus, modelling and analysing message communication latency is critical for optimizing the performance of interprocessor communications. To demonstrate how to achieve efficient (low latency) interprocessor communication, we have developed generalized stochastic Petri net (GSPN) routing models for congestion-prone communication patterns occurring in multicomputer networks. Under congestion-prone adjacent and non-adjacent node communication patterns, we use our developed GSPN routing models to study the effects of congestion (arising due to problems such as buffer overflow and traffic contention) on the message communication latency of multicomputer networks. This modelling has been carried out for mesh multicomputer networks employing the packet switching technique.

Design and evaluation of hardware strategies for reconfiguring hypercubes and meshes under faults

This paper discusses the design of two reconfiguration strategies for distributed memory multicomputer architectures under failures. The specific architectures to which we apply the techniques are hypercubes and meshes. The first scheme uses spare processors attached to certain processors in the hypercube or mash using a novel embedding technique. The second approach places spare processors along specific links in the hypercube or mesh. Both schemes involve the mapping of logical links of a virtual machine onto a set of physical links in the final reconfigured machine and hence suffer some performance degradation. We characterize the performance degradation through trace-driven simulation of real applications running on the faulty and reconfigured system. We find that the schemes have high reliability, suffer little degradation in performance, and are very low in cost. >

Analysis of asynchronous polynomial root finding methods on a distributed memory multicomputer

We have studied various implementations of iterative polynomial root finding methods on a distributed memory multicomputer. These methods are based on the construction of a sequence of approximations that converge to the set of zeros. The synchronous version consists in sharing the computation of the next iterate among the processors and updating their data through a total exchange of their results. In order to decrease the communication cost, we introduce asynchronous versions. The computation of the next iterate is still shared among the processor, but the updating is done by using only nearest neighbor communications. We prove that under weak conditions, these asynchronous versions are still locally convergent, even if their convergence orders are reduced. We analyze the behavior of the asynchronous methods in function of their delay, the topology of the interconnection network, and the elementary computation and communication times. We have implemented and compared these methods on a hypercube multicomputer.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Communication-free data allocation techniques for parallelizing compilers on multicomputers

In distributed memory multicomputers, local memory accesses are much faster than those involving interprocessor communication. For the sake of reducing or even eliminating the interprocessor communication, the array elements in programs must be carefully distributed to local memory of processors for parallel execution. We devote our efforts to the techniques of allocating array elements of nested loops onto multicomputers in a communication-free fashion for parallelizing compilers. We first analyze the pattern of references among all arrays referenced by a nested loop, and then partition the iteration space into blocks without interblock communication. The arrays can be partitioned under the communication-free criteria with nonduplicate or duplicate data. Finally, a heuristic method for mapping the partitioned array elements and iterations onto the fixed-size multicomputers under the consideration of load balancing is proposed. Based on these methods, the nested loops can execute without any communication overhead on the distributed memory multicomputers. Moreover, the performance of the strategies with nonduplicate and duplicate data for matrix multiplication is studied.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Maximum Tolerable Error Bound In Distributed Simulated Annealing

Simulated annealing is an attractive, but expensive, heuristic method for approximating the solution to combinatorial optimization problems. Attempts to parallel simulated annealing, particularly on distributed memory multicomputers, are hampered by the algorithm's requirement of a globally consistent system state. In a multicomputer, maintaining the global state S involves explicit message traffic and is a critical performance bottleneck. To mitigate this bottleneck, it becomes necessary to amortize the overhead of these state updates over as many parallel state changes as possible. By using this technique, errors in the actual cost C(S) of a particular state S will be introduced into the annealing process. This paper places analytically derived bounds on this error in order to assure convergence to the correct optimal result. The resulting parallel simulated annealing algorithm dynamically changes the frequency of global updates as a function of the annealing control parameter, i.e. temperature. Implementation results on an Intel iPSC/2 are reported.

The use of dynamic programming with parallel computers for process synthesis

The generation of optimal heat integrated separation sequences is an important and basic task in chemical process design. The efficient generation of these sequences is crucial because design packages must generate many such sequences. We describe the use of parallel computers to reduce the time required to determine the optimal sequences. In this paper, we concentrate on the use of discrete programming techniques, with particular emphasis on the use of dynamic programming, as implemented on distributed memory multi-computers. We show that, although dynamic programming was superseded for some problems by the use of combined methods including branch and bound for serial machines, a parallel implementation of dynamic programming for process synthesis can be superior.