Distributed Memory Multiprocessor Systems Research Articles

Software implementation of the conjugate gradient method for shared and distributed memory multiprocessor systems

The article examines different approaches to parallelizing the conjugate gradient algorithms for shared and distributed memory multiprocessor systems. We use the standards of MPI and OpenMP libraries of the C programming language for parallel programming. Testing the algorithms was carried out on the HP BLc3000 supercomputer, which contains 392 Intel cores and the InfiniBand network. The article presents the test results – acceleration and efficiency of parallel implementation for each type of multiprocessor systems depending on the dimension of a system of linear equations and the number of parallel processes. We present short descriptions of parallelization methods and excerpts of the code.

Analytical Estimation of the Scalability of Iterative Numerical Algorithms on Distributed Memory Multiprocessors

This article presents a new high-level parallel computational model named BSF - Bulk Synchronous Farm. The BSF model extends the BSP model to deal with the compute-intensive iterative numerical methods executed on distributed-memory multiprocessor systems. The BSF model is based on the master-worker paradigm and the SPMD programming model. The BSF model makes it possible to predict the upper scalability bound of a BSF-program with great accuracy. The BSF model also provides equations for estimating the speedup and parallel efficiency of a BSF-program.

Range query processing on single and multi GPU environments

Metric-space similarity search has proven suitable in a number of application domains such as multimedia retrieval and computational biology to name a few. These applications usually work on very large databases that are often indexed to speed-up on-line searches. To achieve efficient throughput, it is essential to exploit the intrinsic parallelism in the respective search query processing algorithms. Many strategies have been proposed in the literature to parallelize these algorithms either on shared or distributed memory multiprocessor systems. Lately, GPUs have been used to implement brute-force parallel search strategies instead of using index data structures. Indexing poses difficulties when it comes to achieve efficient exploitation of GPU resources. In this paper we propose single and multi GPU metric space techniques that efficiently exploit GPU tailored index data structures for parallel similarity search in large databases. The experimental results show that our proposal outperforms previous index-based sequential and OpenMP search strategies.

Adaptive thermo-fluid moving boundary computations for interfacial dynamics

In this study, we present adaptive moving boundary computation technique with parallel implementation on a distributed memory multi-processor system for large scale thermo-fluid and interfacial flow computations. The solver utilizes Eulerian-Lagrangian method to track moving (Lagrangian) interfaces explicitly on the stationary (Eulerian) Cartesian grid where the flow fields are computed. We address the domain decomposition strategies of Eulerian-Lagrangian method by illustrating its intricate complexity of the computation involved on two different spaces interactively and consequently, and then propose a trade-off approach aiming for parallel scalability. Spatial domain decomposition is adopted for both Eulerian and Lagrangian domain due to easy load balancing and data locality for minimum communication between processors. In addition, parallel cell-based unstructured adaptive mesh refinement (AMR) technique is implemented for the flexible local refinement and even-distributed computational workload among processors. Selected cases are presented to highlight the computational capabilities, including Faraday type interfacial waves with capillary and gravitational forcing, flows around varied geometric configurations and induced by boundary conditions and/or body forces, and thermo-fluid dynamics with phase change. With the aid of the present techniques, large scale challenging moving boundary problems can be effectively addressed.

Parallel WaveCluster: A linear scaling parallel clustering algorithm implementation with application to very large datasets

A linear scaling parallel clustering algorithm implementation and its application to very large datasets for cluster analysis is reported. WaveCluster is a novel clustering approach based on wavelet transforms. Despite this approach has an ability to detect clusters of arbitrary shapes in an efficient way, it requires considerable amount of time to collect results for large sizes of multi-dimensional datasets. We propose the parallel implementation of the WaveCluster algorithm based on the message passing model for a distributed-memory multiprocessor system. In the proposed method, communication among processors and memory requirements are kept at minimum to achieve high efficiency. We have conducted the experiments on a dense dataset and a sparse dataset to measure the algorithm behavior appropriately. Our results obtained from performed experiments demonstrate that developed parallel WaveCluster algorithm exposes high speedup and scales linearly with the increasing number of processors.

Reconfigurable mesh-based inter-chip optical interconnection network for distributed-memory multiprocessor system

A 4×4 reconfigurable mesh-based inter-chip optical interconnection network is reported for distributed-memory multiprocessor system and the experiment confirmed that the data rate in each channel could reach above 3.125 Gbps, which would be a good solution to solve the communication bottlenecks between processors. Each node of this reconfigurable mesh could realize 15 internal connection patterns to complete the interconnections of processors. Besides, this mesh interconnect network via ultra-high bandwidth waveguides embedded in EOPCB can realize flexible multiprocessor system architecture options.

Parallelization of a Monte Carlo particle transport simulation code

We have developed a high performance version of the Monte Carlo particle transport simulation code MC4. The original application code, developed in Visual Basic for Applications (VBA) for Microsoft Excel, was first rewritten in the C programming language for improving code portability. Several pseudo-random number generators have been also integrated and studied. The new MC4 version was then parallelized for shared and distributed-memory multiprocessor systems using the Message Passing Interface. Two parallel pseudo-random number generator libraries (SPRNG and DCMT) have been seamlessly integrated. The performance speedup of parallel MC4 has been studied on a variety of parallel computing architectures including an Intel Xeon server with 4 dual-core processors, a Sun cluster consisting of 16 nodes of 2 dual-core AMD Opteron processors and a 200 dual-processor HP cluster. For large problem size, which is limited only by the physical memory of the multiprocessor server, the speedup results are almost linear on all systems. We have validated the parallel implementation against the serial VBA and C implementations using the same random number generator. Our experimental results on the transport and energy loss of electrons in a water medium show that the serial and parallel codes are equivalent in accuracy. The present improvements allow for studying of higher particle energies with the use of more accurate physical models, and improve statistics as more particles tracks can be simulated in low response time.

Applying Data Mapping Techniques to Vector DSPs

Vector digital signal processors (DSPs) offer a good performance to power consumption ratio. Therefore, they are suitable for mobile devices in software defined radio applications. These vector DSPs require input algorithms with vector operations. The performance of vectorized algorithms to a great extent depends on the distribution of data on vector elements. Traditional algorithms for vectorization focus on the extraction of parallelism from a program; we propose an analysis tool that focuses on the selection of an efficient dynamic data mapping for vector DSPs. We transferred Garcia's communication parallelism graph (Garcia et al., IEEE Trans Parallel Distrib Syst 12: 416---431, 2001) for distributed memory multiprocessor systems to vector DSPs. By alternating the representation of two-dimensional data distributions and the cost models, we are able to determine a dynamic mapping of data on vector elements on the Embedded Vector Processor (EVP) (van Berkel et al., Proceedings of the 2004 software-defined radio technical conference SDR'04, 2004). Additionally, we propose a new efficient algorithm for processing the graph representation that operates in two steps. We demonstrate the capabilities of our tool by describing the vectorization of some MIMO OFDM algorithms.

Numerical simulation of an electromagnetic field by parallelized 3D AIBO-FDTD

The finite-difference time-domain (FDTD) method is useful in solving three-dimensional (3D) electromagnetic problems. However, it contains a numerical instability because it is a complete explicit method, and huge amounts of memory are required in the case of 3D analysis. The implicit FDTD method called the Alternating Implicit Block Overlapped (AIBO)-FDTD method was proposed for two-dimensional (2D) electromagnetic analysis. In our study, the 3D AIBO-FDTD method is developed by the modification of the 2D method. This method is suitable for parallel processing since the computational domain can be effectively partitioned. This property is especially compatible with implementations using distributed-memory multiprocessor systems such as PC clusters, which provide a huge memory space that accommodates a large problem. Although this method seems numerically stable since the implicit scheme is included, our result shows that the numerical instability occurs when the time step exceeds Courant's stability condition. This condition is still more relaxed than the conventional FDTD method. Furthermore, we demonstrated that parallel processing is an effective technique for improving the performance of the 3D AIBO-FDTD method, while this method is slower than FDTD when it is run on one processor.

The performance of parallel iterative solvers

A performance model is constructed for parallel iterative numerical methods under the assumption of a message-passing computing system. Arguments are given for the fact that the speedup of parallel iterative methods is mainly influenced by the speedup at one iterative step. Using the theoretical model, it is proved why explicit iterative methods for ordinary differential equations are inefficient in implementation on distributed memory multiprocessor systems. Numerical tests on parallel and distributed computing environments confirm the correctness of the theoretical model at least in the case of iterative methods for ordinary differential equations and time-dependent partial differential equations.

Investigation on 3-D implicit FDTD method for parallel processing

The finite-difference time-domain (FDTD) method is the well-known method to solve three-dimensional (3-D) electromagnetic problems. However, it contains numerical instability because it is a complete explicit method, and huge memory is required in the case of 3-D analysis. In this paper, the implicit FDTD method called the alternating implicit block overlapped (AIBO)-FDTD method is applied to a 3-D electromagnetic analysis. This method is numerically stable since the implicit scheme is included, and it is suitable for parallel processing since the computational domain can be effectively partitioned. This property is especially compatible with implementations using distributed-memory multiprocessor systems such as PC clusters, which provide a huge memory space that accommodate a large problem. Our results show that parallel processing is an effective technique for improving the performance of the 3-D AIBO-FDTD method, while this method is slower than FDTD when it is run on one processor.

Speedup in solving differential equations on clusters of workstations

A performance model is constructed for parallel iterative numerical methods under the assumption of a message-passing computing system. Arguments are given for the fact that the speedup of parallel iterative methods is mainly influenced by the speedup at one iterative step including computations and communications. Using the theoretical model, it is proved why explicit iterative methods for ordinary differential equations are inefficient in implementation on distributed memory multiprocessor systems. Numerical tests on a cluster of dedicated machines confirm the correctness of the theoretical model at least in the case of iterative methods for ordinary differential equations and time-dependent partial differential equations.

Performance evaluation of a list scheduling algorithm in distributed memory multiprocessor systems

Since it has been shown that the multiprocessor scheduling problem is NP-complete, research efforts have resulted in numerous scheduling algorithms based on heuristics. The need for measuring the effectiveness of the various task scheduling algorithms leads to a great demand on modeling and evaluation tools. This paper proposes to use the stochastic Petri net (SPN) model and its equivalent continuous time Markov chain (CTMC) as the tool for modeling and evaluation of task scheduling algorithms. The proposed approach is applied to DPS (decisive path scheduling) algorithm to investigate its effectiveness. The proposed approach is verified by comparing the result obtained using the SPN model with that obtained by actually running DPS algorithm. The performance comparison reveals that the proposed approach provides very accurate performance evaluation for the scheduling algorithm when the CCR (communication to computation ratio) value is small.

A class of parallel multiple‐front algorithms on subdomains

AbstractA class of parallel multiple‐front solution algorithms is developed for solving linear systems arising from discretization of boundary value problems and evolution problems. The basic substructuring approach and frontal algorithm on each subdomain are first modified to ensure stable factorization in situations where ill‐conditioning may occur due to differing material properties or the use of high degree finite elements (p methods). Next, the method is implemented on distributed‐memory multiprocessor systems with the final reduced (small) Schur complement problem solved on a single processor. A novel algorithm that implements a recursive partitioning approach on the subdomain interfaces is then developed. Both algorithms are implemented and compared in a least‐squares finite‐element scheme for viscous incompressible flow computation using h‐ and p‐finite element schemes. Copyright © 2003 John Wiley & Sons, Ltd.

Task scheduling using a block dependency DAG for block-oriented sparse Cholesky factorization

Block-oriented sparse Cholesky factorization decomposes a sparse matrix into rectangular subblocks; each block can then be handled as a computational unit in order to increase data reuse in a hierarchical memory system. Also, the factorization method increases the degree of concurrency and reduces the overall communication volume so that it performs more efficiently on a distributed-memory multiprocessor system than the customary column-oriented factorization method. But until now, mapping of blocks to processors has been designed for load balance with restricted communication patterns. In this paper, we represent tasks using a block dependency DAG that represents the execution behavior of block sparse Cholesky factorization in a distributed-memory system. Since the characteristics of tasks for block Cholesky factorization are different from those of the conventional parallel task model, we propose a new task scheduling algorithm using a block dependency DAG. The proposed algorithm consists of two stages: early-start clustering, and affined cluster mapping (ACM). The early-start clustering stage is used to cluster tasks while preserving the earliest start time of a task without limiting parallelism. After task clustering, the ACM stage allocates clusters to processors considering both communication cost and load balance. Experimental results on a Myrinet cluster system show that the proposed task scheduling approach outperforms other processor mapping methods.

Indefinitely preconditioned conjugate gradient method for large sparse equality and inequality constrained quadratic problems

AbstractThis paper is concerned with the numerical solution of a symmetric indefinite system which is a generalization of the Karush–Kuhn–Tucker system. Following the recent approach of Lukšan and Vlček, we propose to solve this system by a preconditioned conjugate gradient (PCG) algorithm and we devise two indefinite preconditioners with good theoretical properties. In particular, for one of these preconditioners, the finite termination property of the PCG method is stated. The PCG method combined with a parallel version of these preconditioners is used as inner solver within an inexact Interior‐Point (IP) method for the solution of large and sparse quadratic programs. The numerical results obtained by a parallel code implementing the IP method on distributed memory multiprocessor systems enable us to confirm the effectiveness of the proposed approach for problems with special structure in the constraint matrix and in the objective function. Copyright © 2002 John Wiley & Sons, Ltd.

A framework for integrating data alignment, distribution, and redistribution in distributed memory multiprocessors

Parallel architectures with physically distributed memory provide a cost-effective scalability to solve many large scale scientific problems. However, these systems are very difficult to program and tune. In these systems, the choice of a good data mapping and parallelization strategy can dramatically improve the efficiency of the resulting program. In this paper, we present a framework for automatic data mapping in the context of distributed memory multiprocessor systems. The framework is based on a new approach that allows the alignment, distribution, and redistribution problems to be solved together using a single graph representation. The Communication Parallelism Graph (CPG) is the structure that holds symbolic information about the potential data movement and parallelism inherent to the whole program. The CPG is then particularized for a given problem size and target system and used to find a minimal cost path through the graph using a general purpose linear 0-1 integer programming solver. The data layout strategy generated is optimal according to our current cost and compilation models.

Dynamic task scheduling using online optimization

Algorithms for scheduling independent tasks on to the processors of a multiprocessor system must trade-off processor load balance, memory locality, and scheduling overhead. Most existing algorithms, however, do not adequately balance these conflicting factors. This paper introduces the self-adjusting dynamic scheduling (SADS) class of algorithms that use a unified cost model to explicitly account for these factors at runtime. A dedicated processor performs scheduling in phases by maintaining a tree of partial schedules and incrementally assigning tasks to the least-cost schedule. A scheduling phase terminates whenever any processor becomes idle, at which time partial schedules are distributed to the processors. An extension of the basic SADS algorithm, called DBSADS, controls the scheduling overhead by giving higher priority to partial schedules with more task-to-processor assignments. These algorithms are compared to two distributed scheduling algorithms within a database application on an Intel Paragon distributed memory multiprocessor system.

Optimisation of irregular multiprocessor computer architectures using genetic algorithms

Many problems today need the computing power that is only available by using large‐scaleparallel processing. For a significant number of these problems, the density of theglobal communications between the individual processors dominates the performance ofthe whole parallel implementation on a distributed memory multiprocessor system. In thesecases, the design of the interconnection network for the processors is known to play asignificant part in the efficient implementation of real problems. Important criteria to optimisethe efficiency of a configuration are the maximum and average distance a message hasto travel between processors. Minimum path systems are irregular multiprocessor computerarchitectures which optimise these criteria. These architectures provide an efficient alternativeto the more common regular topologies for solving real applications in parallel. Thispaper presents new results for two combinatorial problems that occur during the generationof these optimal irregular configurations. These are: (1) The design of the optimum interconnection network between the processors for configurationscontaining up to 128 processors. (2) The design of the routing tables to provide the optimal routing of messages withinthese irregular networks. The paper shows how these combinatorial problems have been solved, using genetic algorithmsfor the first problem and a random local search procedure for the second. It alsoincludes a comparison with the results obtained for regular topologies, for example: hypercubes,tori, and rings.

A concurrent network architecture for cost-efficient parallel computing using workstation clusters

Within the past few years, workstation clusters have gained an increasing importance as platforms for parallel high-performance simulation problems. In contrast to the specialized and cost-intensive interconnection network of distributed memory multiprocessor systems, workstation clusters utilize local area networks (LANs) and common communication protocols. Therefore, the cost-efficiency of workstation clusters for parallel tasks is high while the communication performance is limited compared to parallel computer systems. To improve the communication performance of clusters, new protocols can be applied as well as specialized interconnection networks. On one hand, these solutions decrease the cost-efficiency of clusters, while on the other hand the performance of local area networks is increasing because of new technologies such as FastEthernet, GigaEthernet or ATM. In this contribution, we propose increasing the communication performance of clusters through the concurrent network architecture (CNA) with multi-channel communication systems. Through the use of parallel and independent LANs, the communication performance of a cluster can be improved while maintaining the cost-efficiency of the wide-spread LAN technology and protocols. This paper gives an overview of the CNA, the requirements of an implementation and a performance evaluation of a CNA workstation cluster.