Distributed Memory Machines Research Articles

Second-order Møller-Plesset theory with linear R12 terms (MP2-R12) revisited: auxiliary basis set method and massively parallel implementation.

Ab initio electronic structure approaches in which electron correlation explicitly appears have been the subject of much recent interest. Because these methods accelerate the rate of convergence of the energy and properties with respect to the size of the one-particle basis set, they promise to make accuracies of better than 1 kcal/mol computationally feasible for larger chemical systems than can be treated at present with such accuracy. The linear R12 methods of Kutzelnigg and co-workers are currently the most practical means to include explicit electron correlation. However, the application of such methods to systems of chemical interest faces severe challenges, most importantly, the still steep computational cost of such methods. Here we describe an implementation of the second-order Møller-Plesset method with terms linear in the interelectronic distances (MP2-R12) which has a reduced computational cost due to the use of two basis sets. The use of two basis sets in MP2-R12 theory was first investigated recently by Klopper and Samson and is known as the auxiliary basis set (ABS) approach. One of the basis sets is used to describe the orbitals and another, the auxiliary basis set, is used for approximating matrix elements occurring in the exact MP2-R12 theory. We further extend the applicability of the approach by parallelizing all steps of the integral-direct MP2-R12 energy algorithm. We discuss several variants of the MP2-R12 method in the context of parallel execution and demonstrate that our implementation runs efficiently on a variety of distributed memory machines. Results of preliminary applications indicate that the two-basis (ABS) MP2-R12 approach cannot be used safely when small basis sets (such as augmented double- and triple-zeta correlation consistent basis sets) are utilized in the orbital expansion. Our results suggest that basis set reoptimization or further modifications of the explicitly correlated ansatz and/or standard approximations for matrix elements are necessary in order to make the MP2-R12 method sufficiently accurate when small orbital basis sets are used. The computer code is a part of the latest public release of Sandia's Massively Parallel Quantum Chemistry program available under GNU General Public License.

Building with ParadisEO reusable parallel and distributed evolutionary algorithms

Numerous parallel and distributed evolutionary algorithms (PDEAs) and their implementations have been proposed and are available on the Web. A robust approach to make easier their code and design reuse is the framework approach. In this paper, we present some existing frameworks for PDEAs and their development requirements, and propose a new C++ open source framework, named Parallel and distributed Evolving Objects ( ParadisEO). ParadisEO is basically devoted to the reusable and flexible design of parallel and distributed metaheuristics, but we focus here only on PDEAs. Compared to other related frameworks, ParadisEO allows more reuse flexibility, and provides more implemented parallel and distributed models. Furthermore, these models can be exploited by the user in a transparent way, and deployed as well on shared memory multi-processors as on distributed memory machines. The architecture has been experimented on two real-world applications: the radio network design and the spectroscopic data mining. The experimental results demonstrate the efficiency and robustness of the different models.

ParadisEO: A Framework for the Reusable Design of Parallel and Distributed Metaheuristics

In this paper, we present the ParadisEO white-box object-oriented framework dedicated to the reusable design of parallel and distributed metaheuristics (PDM). ParadisEO provides a broad range of features including evolutionary algorithms (EA), local searches (LS), the most common parallel and distributed models and hybridization mechanisms, etc. This high content and utility encourages its use at European level. ParadisEO is based on a clear conceptual separation of the solution methods from the problems they are intended to solve. This separation confers to the user a maximum code and design reuse. Furthermore, the fine-grained nature of the classes provided by the framework allow a higher flexibility compared to other frameworks. ParadisEO is of the rare frameworks that provide the most common parallel and distributed models. Their implementation is portable on distributed-memory machines as well as on shared-memory multiprocessors, as it uses standard libraries such as MPI, PVM and PThreads. The models can be exploited in a transparent way, one has just to instantiate their associated provided classes. Their experimentation on the radio network design real-world application demonstrate their efficiency.

An efficient algorithm for communication set generation of data parallel programs with block-cyclic distribution

Data parallel programming languages, such as High Performance Fortran, are widely regarded as a promising means for writing portable programs for distributed-memory machines. In this paper, we present a new algorithm for computing the communication sets in array section movements with block-cyclic (cyclic (k) in HPF) distribution. Our framework can handle multi-level alignments, multi-dimensional arrays, array intrinsic functions, affine indices and axis exchanges in the array subscript. Instead of employing the linear diophantine equation solver, a new algorithm which does not rely on the linear diophantine equation solver to calculate communication sets is proposed. We use formal proof and experimental results to show that it is more efficient than previous solutions to the same problem. Another important contribution of this paper is that we prove that the compiler is able to compute efficiently the communication sets of block-cyclic distribution as long as the block sizes of the arrays are set to be identical or the lowest common multiple (LCM) of block sizes is not a huge integer. We demonstrate it by thorough complexity analyses and extensive experimental results.

An efficient algorithm for communication set generation of data parallel programs with block-cyclic distribution

Data parallel programming languages, such as High Performance Fortran, are widely regarded as a promising means for writing portable programs for distributed-memory machines. In this paper, we present a new algorithm for computing the communication sets in array section movements with block-cyclic (cyclic ( k) in HPF) distribution. Our framework can handle multi-level alignments, multi-dimensional arrays, array intrinsic functions, affine indices and axis exchanges in the array subscript. Instead of employing the linear diophantine equation solver, a new algorithm which does not rely on the linear diophantine equation solver to calculate communication sets is proposed. We use formal proof and experimental results to show that it is more efficient than previous solutions to the same problem. Another important contribution of this paper is that we prove that the compiler is able to compute efficiently the communication sets of block-cyclic distribution as long as the block sizes of the arrays are set to be identical or the lowest common multiple (LCM) of block sizes is not a huge integer. We demonstrate it by thorough complexity analyses and extensive experimental results.

The local memory access sequence of multiple induction variables on distributed memory machines

Generating the local memory access sequence is an important issue while compiling a data-parallel language into an SPMD code. There are many researches dealing with this issue for cyclic( k) distribution in simple subscripts. But the problem becomes highly complicated when the array references are with complex subscripts. One of the complex array reference functions is array subscript containing multiple induction variables (MIV subscripts). MIV is an affine combination of two or more nested loop indices in a single dimension. In this paper, we develop a general method in O( k) time to evaluate the start element for a given global start element. We compute the course distance first, and then traverse the FSM until the start element is found. In our method, the FSM table constructed in the one-level algorithm [J. Parallel Distr. Comput. 26(1) (1995) 72] can be used directly, but in the method of Ramanujam et al. [Languages and Compilers for Parallel Computing, vol. 1366. Springer-Verlag, 1998, p. 49] the l , r vectors of the processor must be evaluated for each global start element and the time-expensive operations like division and mod are required to check the space boundary of processors. Experimental results have revealed our method is more efficient than theirs.

Group‐SPMD programming with orthogonal processor groups

AbstractMany programs for message‐passing machines can benefit from an implementation in a group‐SPMD programming model due to the potential to reduce communication overhead and to increase scalability. In this paper, we consider group‐SPMD programs exploiting different orthogonal processor partitions in one program. For each program this is a fixed set of predefined processor partitions given by the parallel hyperplanes of a two‐ or multi‐dimensional virtual processor organization. We introduce a library built on top of MPI to support the programming with those orthogonal processor groups. The parallel programming model is appropriate for applications with a multi‐dimensional task grid and task dependencies mainly aligned in the dimensions of the task grid. The library can be used to specify the appropriate processor partitions, which are then created by the library, and to define the mapping of tasks to the processor hyperplanes. Examples from numerical analysis illustrate the programming style and show that the runtime on distributed memory machines can be considerably reduced by using the library. Copyright © 2004 John Wiley & Sons, Ltd.

Flexible parallelization of fast wavelet transforms

In this paper, we present a new parallel algorithm for fast wavelet transforms (FWT) of a matrix of arbitrary size using any given number of parallel processors. The main idea in achieving the optimal load balancing is through a complexity analysis and flops minimization. This makes parallel implementation of FWT feasible and efficient on distributed memory machines with only a small number of processors and on local area networks. The new algorithm is tested by numerical experiments.

A GENERAL PRAM SIMULATION SCHEME FOR CLUSTERED MACHINES

We present a general deterministic scheme to implement a shared memory abstraction on any distributed-memory machine which exhibits a clustered structure. More specifically, we develop a memory distribution strategy and an access protocol for the Decomposable BSP (D-BSP), a generic machine model whose bandwidth/latency parameters can be instantiated to closely reflect the characteristics of machines that admit a hierarchical decomposition into independent clusters. Our scheme achieves provably optimal slowdown for those machines where delays due to latency dominate over those due to bandwidth limitations. For machines where this is not the case, the slowdown is a mere logarithmic factor away from the natural bandwidth-based lower bound.

On approximation of the bulk synchronous task scheduling problem

The bulk synchronous task scheduling problem (BSSPO) is known as an effective task scheduling problem for distributed memory machines. We present a proof of NP-completeness of the decision counterpart of BSSPO, even in the case of makespan of at most five. This implies nonapproximability of BSSPO, meaning that there is no approximation algorithm with performance guarantee smaller than 6/5 unless P = NP. We also give an approximation algorithm with a performance guarantee of two for BSSPO in several restricted cases.

High performance air pollution modeling for a power plant environment

The aim of this work is to provide a high performance air quality simulation using the STEM-II (Sulphur Transport Eulerian Model 2) program, a large-scale pollution modeling application. First, we optimize the sequential program with the aim of increasing data locality. Then, we parallelized the program using OpenMP directives for shared memory systems, and the MPI library for distributed memory machines. Performance results are presented for a SGI O2000 multiprocessor, a Fujitsu AP3000 multicomputer and a Cluster of PCs. Experimental results show that the parallel versions of the code achieve important reductions in the CPU time needed by each simulation. This will allow us to obtain results with adequate speed and reliability for the industrial environment where it is intended to be applied.

Gossiping and broadcasting versus computing functions in networks

In the theory of dissemination of information in interconnection networks (gossiping and broadcasting) one assumes that a message consists of a set of distinguishable, atomic pieces of information, and that one communication pattern is used for solving a task. In this paper, a close connection is established between this theory and a situation in which functions are computed in synchronous networks without restrictions on the type of message used and with possibly different communication patterns for different inputs. The following restriction on the way processors communicate turns out to be essential: (∗) “ Predictable reception”: At the beginning of a step a processor knows whether it is to receive a message across one of its links or not. We show that if (∗) holds then computing an n-ary function with a “critical input” (e.g., the OR of n bits) and distributing the result to all processors on an n-processor network G takes exactly as long as performing gossiping in G. Further we study the complexity of broadcasting one bit in a synchronous network, assuming that in one step a processor can send only one message, but without assuming (∗), and broadcasting one bit on parallel random-access machines (PRAMs) and distributed memory machines (DMMs) with the A RBITRARY access resolution rule.

On variable blocking factor in a parallel dynamic block––Jacobi SVD algorithm

The parallel two-sided block-Jacobi singular value decomposition (SVD) algorithm with dynamic ordering, originally proposed in [Parallel Comput. 28 (2002) 243–262], has been extended with respect to the blocking factor ℓ. Unlike the unique blocking factor ℓ=2 p in the original algorithm running on p processors, the current blocking factor is a variable parameter that covers the values in two different regions––namely, ℓ= p/ k and ℓ=2 kp for some integer k. Two new parallel two-sided block-Jacobi SVD algorithms with dynamic ordering are described in detail. They arise in those two regions and differ in the logical data arrangement and communication complexity of the reordering step. For the case of ℓ=2 kp, it is proved that a designed point-to-point communication algorithm is optimal with respect to the amount of communication required per processor as well as to the amount of overall communication. Using the message passing programming model for distributed memory machines, new parallel block-Jacobi SVD algorithms were implemented on an SGI-Cray Origin 2000 parallel computer. Numerical experiments were performed on p=12 and 24 processors using a set of six matrices of order 4000 and blocking factors ℓ, 2⩽ℓ⩽192. To achieve the minimal total parallel execution time, the use of a blocking factor ℓ∈{2, p,2 p} can be recommended for matrices with distinct singular values. However, for matrices with a multiple minimal singular value, the total parallel execution time may monotonically increase with ℓ. In this case, the recommended Jacobi method with ℓ=2 is just the ScaLAPACK routine with some additional matrix multiplications, and it computes the SVD in one parallel iteration step.

Δ-stepping: a parallelizable shortest path algorithm

The single source shortest path problem for arbitrary directed graphs with n nodes, m edges and nonnegative edge weights can sequentially be solved using O(n· logn+m) operations. However, no work-efficient parallel algorithm is known that runs in sublinear time for arbitrary graphs. In this paper we present a rather simple algorithm for the single source shortest path problem. Our new algorithm, which we call Delta-stepping, can be implemented very efficiently in sequential and parallel setting for a large class of graphs. For random edge weights and arbitrary graphs with maximum node degree d, sequential Δ-stepping needs O(n+m+d·L) total average-case time, where L denotes the maximum shortest path weight from the source node s to any node reachable from s. For example, this means linear time on directed graphs with constant maximum degree. Our best parallel version for a PRAM takes O(d·L· logn+ log 2n) time and O(n+m+d·L· logn) work on average. For random graphs, even O( log 2n) time and O(n+m) work on average can be achieved. We also discuss how the algorithm can be adapted to work with nonrandom edge weights and how it can be implemented on distributed memory machines. Experiments indicate that already a simple implementation of the algorithm achieves significant speedup on real machines.

COMBINING SPECTRAL SEQUENCING AND PARALLEL SIMULATED ANNEALING FOR THE MINLA PROBLEM

In this paper we present and analyze new sequential and parallel heuristics to approximate the Minimum Linear Arrangement problem (MinLA). The heuristics consist in obtaining a first global solution using Spectral Sequencing and improving it locally through Simulated Annealing. In order to accelerate the annealing process, we present a special neighborhood distribution that tends to favor moves with high probability to be accepted. We show how to make use of this neighborhood to parallelize the Metropolis stage on distributed memory machines by mapping partitions of the input graph to processors and performing moves concurrently. The paper reports the results obtained with this new heuristic when applied to a set of large graphs, including graphs arising from finite elements methods and graphs arising from VLSI applications. Compared to other heuristics, the measurements obtained show that the new heuristic improves the solution quality, decreases the running time and offers an excellent speedup when ran on a commodity network made of nine personal computers.

Efficient communication sets generation for block–cyclic distribution on distributed-memory machines

How to generate local memory access sequence and communication sets efficiently is an important issue in compiling a data-parallel language into a single program multiple data (SPMD) code for distributed-memory machines. Many methods have been developed for generating local memory access sequence. In this paper, we focus on the problem of communication sets generation. The local block distance between two active elements with the same offset and destination (source) in a processor will be investigated. We develop an algorithm for the sending phase and receive-execute phase, respectively. Our algorithms do not need to compute send and receive patterns and lose no communication sets while there exist incomplete blocks that cannot constitute a send or receive pattern. Experimental results showed that our method outperforms other previous work.

Symbolic performance modeling of parallel systems

Performance prediction is an important engineering tool that provides valuable feedback on design choices in program synthesis and machine architecture development. We present an analytic performance modeling approach aimed to minimize prediction cost, while providing a prediction accuracy that is sufficient to enable major code and data mapping decisions. Our approach is based on a performance simulation language called PAMELA. Apart from simulation, PAMELA features a symbolic analysis technique that enables PAMELA models to be compiled into symbolic performance models that trade prediction accuracy for the lowest possible solution cost. We demonstrate our approach through a large number of theoretical and practical modeling case studies, including six parallel programs and two distributed-memory machines. The average prediction error of our approach is less than 10 percent, while the average worst-case error is limited to 50 percent. It is shown that this accuracy is sufficient to correctly select the best coding or partitioning strategy. For programs expressed in a high-level, structured programming model, such as data-parallel programs, symbolic performance modeling can be entirely automated. We report on experiments with a PAMELA model generator built within a dataparallel compiler for distributed-memory machines. Our results show that with negligible program annotation, symbolic performance models are automatically compiled in seconds, while their solution cost is in the order of milliseconds.

The parallelized perspective shear-warp algorithm for volume rendering

The shear-warp algorithm for volume rendering is among the fastest volume rendering algorithms. It is an object-order algorithm, based on the idea of the factorization of the view matrix into a 3D shear and a 2D warp component. Thus, the compositing can be done in sheared object space, which allows the algorithm to take advantage of data locality. Although the idea of a perspective projection shear-warp algorithm is not new, it is not widely used so far. That may be because it is slower than the parallel projection algorithm and often slower than hardware supported approaches. In this paper, we present a new parallelized version of the perspective shear-warp algorithm. The parallelized algorithm was designed for distributed memory machines using MPI. The new algorithm takes advantage of the idea that the warp can be done in most computers’ graphics hardware very fast, so that the remote parallel computer only needs to do the compositing. Our algorithm uses this idea to do the compositing on the remote machine, which transfers the resulting 2D intermediate image to the display machine. Even though the display machine can be a mid range PC or laptop computer, it can be used to display complex volumetric data, provided there is a network connection to a high performance parallel computer. Furthermore, remote rendering could be used to drive virtual environments, which typically require perspective projection and high frame rates for stereo projection and multiple screens.

Optimization of Data Distribution and Processor Allocation Problem Using Simulated Annealing

In this study, a global optimization meta-heuristic is developed for the problem of determining the optimum data distribution and degree of parallelism in parallelizing a sequential program for distributed memory machines. The parallel program is considered as the union of consecutive stages and the method deals with all the stages in the entire program rather than proposing solutions for each stage. The meta-heuristic developed here for this specific problem combines simulated annealing and hill climbing (SA-HC) in the search for the optimum configuration. Performance is tested in terms of the total execution time of the program including communication and computation times. Two exemplary codes from the literature, the first being computation intensive and the second being communication intensive, are utilized in the experiments. The performance of the SA-HC algorithm provides satisfactory results for these illustrative examples.

Parallel Complexity of Matrix Multiplication

Effective design of parallel matrix multiplication algorithms relies on the consideration of many interdependent issues based on the underlying parallel machine or network upon which such algorithms will be implemented, as well as, the type of methodology utilized by an algorithm. In this paper, we determine the parallel complexity of multiplying two (not necessarily square) matrices on parallel distributed-memory machines and/or networks. In other words, we provided an achievable parallel run-time that can not be beaten by any algorithm (known or unknown) for solving this problem. In addition, any algorithm that claims to be optimal must attain this run-time. In order to obtain results that are general and useful throughout a span of machines, we base our results on the well-known LogP model. Furthermore, three important criteria must be considered in order to determine the running time of a parallel algorithm; namely, (i) local computational tasks, (ii) the initial data layout, and (iii) the communication schedule. We provide optimality results by first proving general lower bounds on parallel run-time. These lower bounds lead to significant insights on (i)–(iii) above. In particular, we present what types of data layouts and communication schedules are needed in order to obtain optimal run-times. We prove that no one data layout can achieve optimal running times for all cases. Instead, optimal layouts depend on the dimensions of each matrix, and on the number of processors. Lastly, optimal algorithms are provided.