Heterogeneous Workstation Clusters Research Articles

Heterogeneous parallel computing: from clusters of workstations to hierarchical hybrid platforms

The paper overviews the state of the art in design and implementation of data parallel scientific applications on heterogeneous platforms. It covers both traditional approaches originally designed for clusters of heterogeneous workstations and the most recent methods developed in the context of modern multicore and multi-accelerator heterogeneous platforms.

The parallelization of the Keller box method on heterogeneous cluster of workstations

High performance computing is the branch of parallel computing dealing with very large problems and large parallel computers that can solve those problems in a reasonable amount of time. This paper will describe the parallelization of the Keller-box method using the high performance computing on heterogeneous cluster of workstations. The problem statement is based on the equation of boundary-layer flow due to a moving flat plate. The objective is to develop the parallel algorithm of the Keller-box method in purpose to solve a large size of matrix. The parallelization is based on the domain decomposition, where the upper and lower matrices will be splitting into a number of blocks, which then will be compute concurrently on the parallel computers. The experiment was run using 200, 2000, and 20000 size of matrices and using 10 number of processors. The comparison was made from the results obtained from that various size of matrices by doing the analysis based on the performance measurement in terms of time execution, speedup, and effectiveness.

COMPUTING DOT -- PRODUCT ON HETEROGENEOUS MASTER -- WORKER PLATFORMS

This paper is focused on designing two parallel dot product imple- mentations for heterogeneous master-worker platforms. These implementations are based on the data allocation and dynamic load balancing strategies. The first implementation is the dynamic master - worker with allocation of vectors where the master distributes vectors (data) and computations to the workers whereas the second one is the dynamic master - worker with allocation of vector pointers where the vectors are supposed to be replicated among participating resources beforehand and the master distributes computations to the workers. We also report a set of numerical experiments on a heterogeneous platform where computational resources have different computing powers. Also, the workers are connected to the master by links of same capacities. The obtained experimental results demonstrate that the dynamic allocation of vector point- ers achieve better performance than the original one for computing dot product computation. The paper also presents and verifies an accurate timing model to predict the performance of the proposed implementations on clusters of heterogeneous workstations. Through this model the viability of the proposed implementa-

Load balancing on temporally heterogeneous cluster of workstations for parallel simulated annealing

Simulated annealing (SA) is a general-purpose optimization technique widely used in various combinatorial optimization problems. However, the main drawback of this technique is a long computation time required to obtain a good quality of solution. Clusters have emerged as a feasible and popular platform for parallel computing in many applications. Computing nodes on many of the clusters available today are temporally heterogeneous. In this study, multiple Markov chain (MMC) parallel simulated annealing (PSA) algorithms have been implemented on a temporally heterogeneous cluster of workstations to solve the graph partitioning problem and their performance has been analyzed in detail. Temporal heterogeneity of a cluster of workstations is harnessed by employing static and dynamic load balancing techniques to further improve efficiency and scalability of the MMC PSA algorithms.

Explicit nonlinear dynamic finite element analysis on homogeneous/heterogeneous parallel computing environment

This paper presents parallel computational strategies to implement explicit nonlinear finite element analysis code onto distributed memory parallel computers for solving large-scale problems in structural dynamics. Implementation details on both homogeneous and heterogeneous parallel processing environments are considered in detail in this paper. Implementation of an explicit nonlinear finite element dynamic analysis code on homogeneous systems is discussed first and this is later moved onto heterogeneous systems. Domain decomposition with explicit message passing is preferred for parallel implementation. The message passing implementation in the parallel algorithm is based on MPI (Message Passing Interface) libraries. Implementation aspects of overlapped, non-overlapped domain decomposition techniques, Dynamic Task Allocation (DTA) and clustering techniques for DTA and their relative merits are presented. The interprocessor communications are optimised by overlapping with computations to improve the performance of the domain decomposition based explicit dynamic analysis finite element code. The issues related to implementation of finite element code for nonlinear dynamic analysis on heterogeneous parallel computing environment are later presented. A new dynamic load-balancing algorithm is developed for this purpose and it is integrated with the domain decomposition based parallel explicit finite element code to test our algorithms on a coarse grain heterogeneous cluster of workstations. Numerical experiments have been carried out on PARAM-10000, an Indian parallel computer and also on cluster of Unix workstations.

Parallel text searching applications on a heterogeneous cluster architecture

In this paper, we propose four text searching implementations on a cluster of heterogeneous workstations using MPI message passing library. The first three parallel implementations are based on the static and dynamic master-worker methods. Further, we propose a hybrid parallel implementation that combines the advantages of static and dynamic parallel methods in order to reduce the load imbalance and communication overhead. We test these parallel implementations and present experimental results for different text sizes and number of workstations. We also propose a performance prediction model of the four implementations that agrees well with our experimental measurements.

Component Object Based Single System Image for Dependable Implementation of Genetic Programming on Clusters

We present a distributed component-object model (DCOM) based single system image (SSI) for dependable parallel implementation of genetic programming (DPIGP). DPIGP is aimed to significantly and reliably improve the computational performance of genetic programming (GP) exploiting the inherent parallelism in GP among the evaluation of individuals. It runs on cost-effective clusters of commodity, non-dedicated, heterogeneous workstations or PCs. Developed SSI represents the pool of heterogeneous workstations as a single, unified virtual resource – a metacomputer, and addresses the issues of locating and allocating the physical resources, communicating between the entities of DPIGP, scheduling and load balancing. In addition, addressing the issue of fault tolerance, SSI allows for building a highly available metacomputer in which the cases of workstation failure result only in a corresponding partial degradation of the overall performance characteristics of DPIGP. Adopting DCOM as a communicating paradigm offers the benefits of software platform- and network protocol neutrality of proposed approach; and the generic support for the issues of locating, allocating and security of the distributed entities of DPIGP.

Performance evaluation of load balancing strategies for approximate string matching application on an MPI cluster of heterogeneous workstations

In this paper, we present three parallel approximate string matching methods on a parallel architecture with heterogeneous workstations to gain supercomputer power at low cost. The first method is the static master–worker with uniform distribution strategy, the second one is the dynamic master–worker with allocation of subtexts and the third one is the dynamic master–worker with allocation of text pointers. Further, we propose a hybrid parallel method that combines the advantages of static and dynamic parallel methods in order to reduce the load imbalance and communication overhead. This hybrid method is based on the following optimal distribution strategy: the text collection is distributed proportional to workstation’s speed. We evaluated and compared the performance of the four methods with clusters one, two, four, six and eight heterogeneous workstations. The experimental results demonstrate that dynamic allocation of text pointers and hybrid methods achieve better performance than the two original ones. We also present an analytical performance model for the four methods that confirms the actual behaviour of the experimental results.

Homogeneous and heterogeneous distributed cluster processing for two‐ and three‐dimensional viscoelastic flows

AbstractA finite‐element study of two‐ and three‐dimensional incompressible viscoelastic flows in a planar lid‐driven cavity and concentric rotating cylinders is presented. The hardware platforms consist of both homogeneous and heterogeneous clusters of workstations. A semi‐implicit time‐stepping Taylor–Galerkin scheme is employed using the message passing mechanism provided by the Parallel Virtual Machine libraries. DEC‐alpha, Intel Solaris and AMD‐K7(Athlon) Linux clusters are utilized. Parallel results are compared against single processor (sequentially) solutions, using the parallelism paradigm of domain decomposition. Communication is effectively masked and practically ideal, linear speed‐up with the number of processors is realized. Copyright © 2002 John Wiley & Sons, Ltd.

Parallelizing Tabu Search on a Cluster of Heterogeneous Workstations

In this paper, we present the parallelization of tabu search on a network of workstations using PVM. Two parallelization strategies are integrated: functional decomposition strategy and multi-search threads strategy. In addition, domain decomposition strategy is implemented probabilistically. The performance of each strategy is observed and analyzed. The goal of parallelization is to speedup the search in finding better quality solutions. Observations support that both parallelization strategies are beneficial, with functional decomposition producing slightly better results. Experiments were conducted for the VLSI cell placement, an NP-hard problem, and the objective was to achieve the best possible solution in terms of interconnection length, timing performance (circuit speed), and area. The multiobjective nature of this problem is addressed using a fuzzy goal-based cost computation.

MPI–Delphi: an MPI implementation for visual programming environments and heterogeneous computing

The goal of a parallel program is to reduce the execution time, compared to the fastest sequential program solving the same problem. Parallel programming is growing due to the widespread use of network of workstations (NOWs) or powerful PCs in high-performance computing. Because the hardware components are all commodity devices, NOWs are much more cost-effective than custom machines with similar technology. In this environment, the typical programming model used has been message-passing and the MPI library has become the standard in the distributed-memory computing model. On the other hand, visual programming environments try to simply the task of developing applications. They provide programmers with several standard components for creating programs. Delphi constitutes one of the most popular visual programming environments nowadays in the Windows market place. In this paper, we present MPI–Delphi, an implementation of MPI for writing parallel applications using Delphi visual programming environment. We show how MPI–Delphi has been developed, and how it makes possible to manage a cluster of homogeneous/heterogeneous PCs. Two examples of use of MPI–Delphi in a heterogeneous cluster of workstations with a mixture of Windows and Linux operating systems are also included. The MPI–Delphi interface is suitable for some specific kinds of problems, such as monitoring parallel programs of long execution time, or computationally intensive graphical simulations. In addition, MPI–Delphi has proven to be a good tool for research, as the development of new algorithms can be carried out quickly and, therefore, time spent on the debugging of such algorithms is reduced. Finally, we conclude by explaining some of the tasks we think MPI–Delphi is suitable for.

Development and performance analysis of real‐world applications for distributed and parallel architectures

AbstractSeveral large real‐world applications have been developed for distributed and parallel architectures. We examine two different program development approaches. First, the usage of a high‐level programming paradigm which reduces the time to create a parallel program dramatically but sometimes at the cost of a reduced performance; a source‐to‐source compiler, has been employed to automatically compile programs—written in a high‐level programming paradigm—into message passing codes. Second, a manual program development by using a low‐level programming paradigm—such as message passing—enables the programmer to fully exploit a given architecture at the cost of a time‐consuming and error‐prone effort. Performance tools play a central role in supporting the performance‐oriented development of applications for distributed and parallel architectures. SCALA—a portable instrumentation, measurement, and post‐execution performance analysis system for distributed and parallel programs—has been used to analyze and to guide the application development, by selectively instrumenting and measuring the code versions, by comparing performance information of several program executions, by computing a variety of important performance metrics, by detecting performance bottlenecks, and by relating performance information back to the input program. We show several experiments of SCALA when applied to real‐world applications. These experiments are conducted for a NEC Cenju‐4 distributed‐memory machine and a cluster of heterogeneous workstations and networks. Copyright © 2001 John Wiley & Sons, Ltd.

Object Placement Using Performance Surfaces

Heterogeneous parallel clusters of workstations are being used to solve many important computational problems. Scheduling parallel applications on the best collection of machines in a heterogeneous computing environment is a complex problem. Performance prediction is vital to good application performance in this environment since utilization of an ill-suited machine can slow the computation down significantly. This paper addresses the problem of network performance prediction. A new methodology for characterizing network links and application's need for network resources is developed which makes use of Performance Surfaces [3]. This Performance Surface abstraction is used to schedule a parallel application on resources where it will run most efficiently.

Fault-Tolerant Parallel Scheduling of Tasks on a Heterogeneous High-Performance Workstation Cluster

We propose a new approach, called cluster-based search (CBS), for scheduling large task graphs in parallel on a heterogeneous cluster of workstations connected by a high-speed network (e.g., using an ATM switch at OC-3 speed). The CBS algorithm uses a parallel random neighborhood search which works by refining multiple different initial schedules simultaneously using different workstations. The workstations communicate periodically to exchange their best solutions found thus far in order to direct the search to more promising regions in the search space. Heterogeneity of machines is exploited by the biased partitioning of the search space. The parallel random neighborhood search is fault-tolerant in that the workload of a failed workstation is automatically redistributed to other workstations so that the search can continue. We have implemented the CBS algorithm as a core function of our on-going development of SSI middleware for a Sun workstation cluster.

EDPEPPS: an integrated environment for the parallel development life-cycle

This paper describes an environment for performance-oriented design of portable parallel software. The environment consists of a graphical design tool for building parallel algorithms, a state-of-the-art simulation engine, a CPU characterisation tool, a distributed debugging tool and a visualisation/replay tool. The environment is used to model a virtual machine composed of a cluster of heterogeneous workstations interconnected by a local area network. The simulation model used is modular and its components are interchangeable which allows easy re-configuration of the platform. The model is validated using experiments on two parallel Givens linear solver algorithms with average errors of about 8%.

Parallelizing the Dual Simplex Method

We study the parallelization of the steepest-edge version of the dual simplex algorithm. Three different parallel implementations are examined, each of which is derived from the CPLEX dual simplex implementation. One alternative uses PVM, one general-purpose System V shared-memory constructs, and one the PowerC extension of C on a Silicon Graphics multi-processor. These versions were tested on different parallel platforms, including heterogeneous workstation clusters, Silicon Graphics multi-processors, and an IBM SP2. We report on our computational experience.

Distributed computing for membrane-based modeling of action potential propagation.

Action potential propagation simulations with physiologic membrane currents and macroscopic tissue dimensions are computationally expensive. We, therefore, analyzed distributed computing schemes to reduce execution time in workstation clusters by parallelizing solutions with message passing. Four schemes were considered in two-dimensional monodomain simulations with the Beeler-Reuter membrane equations. Parallel speedups measured with each scheme were compared to theoretical speedups, recognizing the relationship between speedup and code portions that executed serially. A data decomposition scheme based on total ionic current provided the best performance. Analysis of communication latencies in that scheme led to a load-balancing algorithm in which measured speedups at 89 +/- 2% and 75 +/- 8% of theoretical speedups were achieved in homogeneous and heterogeneous clusters of workstations. Speedups in this scheme with the Luo-Rudy dynamic membrane equations exceeded 3.0 with eight distributed workstations. Cluster speedups were comparable to those measured during parallel execution on a shared memory machine.

Chronos: a Performance Characterization Tool Inside the EDPEPPS Toolset

The EDPEPPS toolset is the fruit of a 10 man-year-research development and integrates many modules in order to predict and classify the execution times of C/PVM programs mapped on a cluster of heterogeneous workstations. In this project, a performance characterization tool called Chronos has been developed to model the processor and C instructions. Chronos can be used to characterize a wide range of machines as it is developed round a specialized benchmark. Chronos uses a parameter-based model and characterizes the machine and the program studied. Then, the execution predictor evaluates the time spent in each program block, according to a generic model of cache memory which simulates most of the CPU internal cache memory architecture. Chronos does not need any user's intervention as all the operations are automatic. The performance accuracy of Chronos is highlighted by a real processor-consuming sequential example. This tool can be used by designers to predict the average execution time of their applications quickly. Average percentage errors obtained from this tool are below 10%.

Design and Implementation of DynamicLoad Balancing Algorithms for Rollback Reductionin Optimistic PDES

In an optimistic parallel simulation, logical processes (Ips) proceed with their computation without any constraints. However, if the computing requirements of different lps are not balanced or if the processors are not homogeneous, some lps may lag behind in simulation time while others surge forward. In other words, if the simulation clocks of different lps are not progressing at the same rate, cascading rollbacks may occur nullifying the potential benefit of an optimistic parallel discrete event simulation (PDES). Hence it is necessary to balance the computational load on different lps in such a way that their local simulation clocks advance almost at the same rate. In this paper, we propose two algorithms for dynamic load balancing which reduce the number of rollbacks in an optimistic PDES system. Our first algorithm is based on the load transfer mechanism between lps; while the second algorithm, based on the principle of evolutionary strategy, migrates logical processes between several pairs of physical processors. We have implemented both of these algorithms on a cluster of heterogeneous workstations and studied their performance. The experimental results show that the algorithm based on the load transfer is effective when the grain size is greater than 10 milliseconds. The algorithm based on the process migration yields good performance only for grain sizes of 20 milliseconds or larger. In both of these cases the speed up ranges mostly between and 2 using four processors.

The effectiveness of domain balancing strategies on workstation clusters demonstrated by viscous flow problems

We consider several aspects of efficient numerical simulation of viscous compressible flow on both homogeneous and heterogeneous workstation-clusters. We consider dedicated systems, as well as clusters operating in a multi-user environment. For dedicated homogeneous clusters, we show that with respect to the turn-around time, there is an optimal number of workstations to solve a given flow-problem. This number depends on the actual implementation of the solver. For non-dedicated heterogeneous clusters we apply dynamic domain-balancing techniques in order to obtain an optimal turn-around time for the flow-simulation job only, taking into account the activities on the cluster arising from the other applications. We show that the decision which technique should be used depends on various aspects, such as the character of the load-fluctuations arising from the other applications, whether the flow-simulation job is computationally intensive, and the underlying hardware. The effects of these aspects on this decision are analyzed. Although it can be concluded that no domain-balancing technique is the best under arbitrary circumstances, such an analysis may guide the decision for a particular load-situation. Moreover, we indicate how such a decision can be supported by a comparison of the various balancing strategies using a simulation study. This is illustrated for a specific load-situation.