Superlinear Speedup Research Articles

Servo

Conventional programming models were designed to be used by expert programmers for programming for large-scale multiprocessors, distributed computational clusters, or specialized parallel machines. These models, therefore, are deemed either too difficult for an "average" programmer (who will be expected to do parallel programming in the many-core world) or too inefficient to use for many-core architectures [3]. Similarly, conventional execution models were designed for performance, not scalability. To address the challenge of performance scalability for many-core architectures, we introduce Servo, a Service Oriented programming model that decomposes every program into a set of components (each with its own local memory, mutable state, and a program counter) that either request services or deliver services. Servo is characterized by the decoupling of logical communication mapping between program modules (or services) from physical communication mapping between modules. This allows the services to be migrated and replicated during execution. The proposed model also allows the granularity of data parallel operations to be changed dynamically. This allows runtime variation in the locking granularity which in turn enables higher write-parallelism. Finally, the models partition even data into services. This can significantly enhance locality and makes it possible to have superlinear speedups with increasing number of cores. Our preliminary investigations demonstrate significant performance scalability advantages for Servo.

A prototype implementation of a distributed satisfiability modulo theories solver in the ToolBus framework

An increasing number of verification tools (e.g., software model-checkers) require the use of Satisfiability Modulo Theories (SMT) solvers to implement the back-ends for the automatic analysis of specifications and properties. The most prominent approach to build SMT solvers consists in integrating an efficient Boolean solver with decision procedures capable of checking the satisfiability of sets of ground literals in selected theories. Although the problem of checking the satisfiability of arbitrary Boolean combinations of atoms modulo a background theory is NP-hard, there is a strong demand for high-performance SMT-solvers. In this paper, we describe the design and prototype implementation of-to the best of our knowledge-the first distributed SMT solver. The emphasis is on providing ways to reduce the implementation effort and to make the system easily extensible. This is achieved in two ways: (a) we re-use as much as possible the code of an available sequential SMT solver and (b) we adopt the TOOLBUS architecture for rapid prototyping. The behavior of the distributed SMT solver was tested on a set of problems which are representative of those generated by software verification techniques. The experiments show the possibility to obtain super-linear speed-ups of the distributed SMT solver with respect to its sequential version.

A prototype implementation of a distributed Satisfiability Modulo Theories solver in the ToolBus framework

Abstract An increasing number of verification tools (e.g., software model-checkers) require the use of Satisfiability Modulo Theories (SMT) solvers to implement the back-ends for the automatic analysis of specifications and properties. The most prominent approach to build SMT solvers consists in integrating an efficient Boolean solver with decision procedures capable of checking the satisfiability of sets of ground literals in selected theories. Although the problem of checking the satisfiability of arbitrary Boolean combinations of atoms modulo a background theory is NP-hard, there is a strong demand for high-performance SMT-solvers. In this paper, we describe the design and prototype implementation of—to the best of our knowledge—the first distributed SMT solver. The emphasis is on providing ways to reduce the implementation effort and to make the system easily extensible. This is achieved in two ways: (a) we re-use as much as possible the code of an available sequential SMT solver and (b) we adopt theToolBus architecture for rapid prototyping. The behavior of the distributed SMT solver was tested on a set of problems which are representative of those generated by software verification techniques. The experiments show the possibility to obtain super-linear speed-ups of the distributed SMT solver with respect to its sequential version.

A Parallel Genetic Algorithm for the Multilevel Unconstrained Lot-Sizing Problem

Acoarse-grained parallel genetic algorithm (PGA) for the multilevel unconstrained lot-sizing problem (MLULSP) is described and evaluated using 176 benchmark problems from the literature, with problem sizes varying from 5 to 500 products and with simulations ranging from 12 to 52 periods. The parallelization approach consists of concurrently executing several subpopulations (demes) of a genetic algorithm with the occasional migration of individuals between them. The migration is controlled by the migration model of Cantú-Paz [Cantú-Paz, E. 2001. Migration policies, selection pressure, and parallel evolutionary algorithms. J. Heuristics 7 311–334]. The genetic algorithm used is based on a new binary coding for the MLULSP. The PGA, which is the first application of a parallel metaheuristic to the MLULSP, is competitive with the best known solution methods. It was possible with the new method to calculate new best solutions for 34 of the benchmark problems. Furthermore, the migration enables a super-linear speedup. The results indicate that parallel genetic algorithms are suitable for solving large-problem instances of the MLULSP with acceptable computing times.

Adaptive hybrid partitioning for OLAP query processing in a database cluster

We consider the use of a database cluster for high-performance support of Online Analytical Processing (OLAP) applications. OLAP intra-query parallelism can be obtained by partitioning the database tables across cluster nodes. We propose to combine physical and virtual partitioning into a partitioning scheme called Adaptive Hybrid Partitioning (AHP). AHP requires less disk space while allowing for load balancing. We developed a prototype for OLAP parallel query processing in database clusters using AHP. Our experiments on a 32-node database cluster using the TPC-H benchmark demonstrate linear and super-linear speedup. Thus, AHP can reduce significantly the execution time of typical OLAP queries.

Tashkent+

We present a memory-aware load balancing (MALB) technique to dispatch transactions to replicas in a replicated database. Our MALB algorithm exploits knowledge of the working sets of transactions to assign them to replicas in such a way that they execute in main memory, thereby reducing disk I/O. In support of MALB, we introduce a method to estimate the size and the contents of transaction working sets. We also present an optimization called update filtering that reduces the overhead of update propagation between replicas. We show that MALB greatly improves performance over other load balancing techniques -- such as round robin, least connections, and locality-aware request distribution (LARD) -- that do not use explicit information on how transactions use memory. In particular, LARD demonstrates good performance for read-only static content Web workloads, but it gives performance inferior to MALB for database workloads as it does not efficiently handle large requests. MALB combined with update filtering further boosts performance over LARD. We build a prototype replicated system, called Tashkent+, with which we demonstrate that MALB and update filtering techniques improve performance of the TPC-W and RUBiS benchmarks. In particular, in a 16-replica cluster and using the ordering mix of TPC-W, MALB doubles the throughput over least connections and improves throughput 52% over LARD. MALB with update filtering further improves throughput to triple that of least connections and more than double that of LARD. Our techniques exhibit super-linear speedup; the throughput of the 16-replica cluster is 37 times the peak throughput of a standalone database due to better use of the cluster's memory.

A Detailed Performance Analysis of the Interpolation Supplemented Lattice Boltzmann Method on the Cray T3E and Cray X1A Detailed Performance Analysis of the Interpolation Supplemented Lattice Boltzmann Method on the Cray T3E and Cray X1

A detailed study of the parallel performance of the interpolation supplemented lattice Boltzmann (ISLB) method using SHMEM and MPI on the Cray T3E-900 and Cray X1 architectures is presented. The noteworthy feature of the present implementation of the ISLB method is that it is able to achieve a sustained speed of 4.2 Tflop/s while using 504 processors on a Cray X1. The code is shown to achieve super-linear speedups on the Cray T3E-900. It is shown through detailed profiling that the computation and the communication scale well on the Cray X1, although the overall speedup is adversely affected by the cost of barrier synchronization.

MPI framework for parallel searching in large biological databases

In this paper, we address the problem of searching huge biological databases on the scale of at least several gigabytes by utilizing parallel processing. Biological databases storing DNA sequences, protein sequences, or mass spectra are growing exponentially. Searches through these databases consume exponentially growing computational resources as well. We demonstrate herein a general use, MPI based, C ++ framework for generically splitting databases amongst several computational nodes. The combined RAM of the nodes working in tandem is often sufficient to keep the entire database in memory, and therefore to search it efficiently without paging to disk. The framework runs as a persistent service, processing all submitted queries. This allows for query reordering and better utilization of the memory. Thereby, we achieve superlinear speedups compared to single processor implementations. We demonstrate the utility and speedup of the framework using a real biological database and an actual searching algorithm for mass spectrometry.

Parallel network simulations with NEURON.

The NEURON simulation environment has been extended to support parallel network simulations. Each processor integrates the equations for its subnet over an interval equal to the minimum (interprocessor) presynaptic spike generation to postsynaptic spike delivery connection delay. The performance of three published network models with very different spike patterns exhibits superlinear speedup on Beowulf clusters and demonstrates that spike communication overhead is often less than the benefit of an increased fraction of the entire problem fitting into high speed cache. On the EPFL IBM Blue Gene, almost linear speedup was obtained up to 100 processors. Increasing one model from 500 to 40,000 realistic cells exhibited almost linear speedup on 2,000 processors, with an integration time of 9.8 seconds and communication time of 1.3 seconds. The potential for speed-ups of several orders of magnitude makes practical the running of large network simulations that could otherwise not be explored.

A Parallel Solution to the Radiative Transport in Three-Dimensional Participating Media

The discrete ordinates method as a solution to the radiative transport equation is parallelized using the domain decomposition approach for three-dimensional Cartesian geometries with a diffusively reflecting wall. The method contains absorbing, emitting, and anisotropically scattering media. We investigate load balancing and define a pipelining procedure. The algorithm is tested on a cluster of personal computers, and super linear speed-up is observed for cases of a certain data size, as a result of the cache optimization of the compiler.

INHERENTLY PARALLEL GEOMETRIC COMPUTATIONS

A new computational paradigm is described which offers the possibility of superlinear (and sometimes unbounded) speedup, when parallel computation is used. The computations involved are subject only to given mathematical constraints and hence do not depend on external circumstances to achieve superlinear performance. The focus here is on geometric transformations. Given a geometric object A with some property, it is required to transform A into another object B which enjoys the same property. If the transformation requires several steps, each resulting in an intermediate object, then each of these intermediate objects must also obey the same property. We show that in transforming one triangulation of a polygon into another, a parallel algorithm achieves a superlinear speedup. In the case where a convex decomposition of a set of points is to be transformed, the improvement in performance is unbounded, meaning that a parallel algorithm succeeds in solving the problem as posed, while all sequential algorithms fail.

Delaunay Decoupling Method for Parallel Guaranteed Quality Planar Mesh Refinement

Creating in parallel guaranteed quality large unstructured meshes is a challenging problem. Parallel mesh generation procedures decompose the original mesh generation problem into smaller subproblems that can be solved in parallel. The subproblems can be treated as either completely or partially coupled, or they can be treated as completely decoupled. In this paper we present a parallel guaranteed quality Delaunay method for 2-dimensional domains which is based on the complete decoupling of the subproblems. As a result the method eliminates the communication and the synchronization during the meshing of the subproblems. Moreover, it achieves 100% code reuse of existing, fine-tuned, and well-tested sequential mesh generators. The approach we describe in this paper presents for the first time an effective way to create in parallel guaranteed quality meshes with billions of elements in a few hundreds of seconds, and at the same time demonstrates that these meshes can be generated in an efficient and scalable way. Our performance data indicate superlinear speedups.

VoD on steroids : Optimized content delivery using distributed video servers over best-effort Internet

This paper addresses the problem of delivering a long-duration continuous media document, typically a movie, over unreliable communication links like the ones that make up the Internet. Packet loss/drop is one of the main problems that plague Internet's use for offering multimedia services. The framework presented here addresses analytically this problem by offering closed-form solutions for partitioning and scheduling the delivery of a document by multiple spatially distributed servers. The goal is to offer uninterrupted playback to clients while minimizing their access-time. The solid mathematical foundation presented, enables the study of multi-server document delivery using unreliable communications for both CBR and VBR media. It is also shown how slow connections that would otherwise prevent the deployment of Movie-on-Demand class services, could be used to offer such services in an optimal manner. The paper also addresses the problem of optimally arranging the available servers. A rigorous simulation study that explores the potential of the proposed approach concludes the paper. Our simulations show a super-linear speedup in access-times and also the robustness of the proposed scheme, capable of handling unknown network conditions by shifting server “load”.

A high-performance distributed algorithm for mining association rules

We present a new distributed association rule mining (D-ARM) algorithm that demonstrates superlinear speedup with the number of computing nodes. The algorithm is the first D-ARM algorithm to perform a single scan over the database. As such, its performance is unmatched by any previous algorithm. Scale-up experiments over standard synthetic benchmarks demonstrate stable run time regardless of the number of computers. Theoretical analysis reveals a tighter bound on error probability than the one shown in the corresponding sequential algorithm.

A parallel and scalable algorithm for ADI method with pre-propagation and message vectorization

It is hard to implement the ADI method in an efficient way on distributed-memory parallel computers. We propose “ P-scheme” which parallelizes a tridiagonal linear system of equations for the ADI method, but its effectiveness is limited to the cases where the problem size is large enough mainly because of the communication cost of the propagation phase of the scheme. In order to overcome this difficulty, we propose an improved version of the P-scheme with “ message vectorization” which aggregates several communication messages into one and alleviates the communication cost. Also we evaluate the effectiveness of message vectorization for the ADI method and show that the improved version of the P-scheme works well even for smaller problems and linear and super-linear speedups can be achieved for 8194 × 8194 and 16,386 × 16,386 problems, respectively.

Practically oriented parallel Delaunay triangulation in E 2 for computers with shared memory

This paper describes two simple and efficient parallel algorithms for the construction of the Delaunay triangulation ( DT( S)) in E 2 by randomized incremental insertion. The construction of the DT( S) is one of the fundamental problems in computer graphics. The proposed algorithms are designed for parallel systems several processors and with shared memory. Such a hardware configuration (especially the case with two-processors) became widely available in the last few years thanks to low prices at present, but there is still a lack of parallel algorithms that are simple to implement and efficient enough to be an attractive alternative to existing serial algorithms. We have implemented both new algorithms in C++ and tested them on workstations with up to four processors. Thanks to memory caching we noticed several times even super-linear speed-up compared with the reference sequential algorithm.

Solving Eigenproblems on Multicomputers: Two Different Approaches

The authors analyze two direct methods for solving the symmetric eigenproblem of large and sparse matrices in terms of their parallel implementations. Their proposals provide efficient parallel solutions for problems with large computational requirements. The direct solutions were decomposed in the following phases: (1) structuring the input matrix — the Lanczos method with complete reorthogonalization was implemented for this stage; (2) solving the eigenproblem of a structured matrix — two methods were applied to solve this stage, bisection and inverse iteration methods, and the divide-and-conquer method; (3) computing the eigenvectors of the input matrix, carried out by a matrix–matrix product. These methods have been implemented on a multiprocessor system and their performance evaluations carried out on a Cray T3E system with up to 32 nodes. Results show tha t the management of the memory hierarchy improves substantially as the number of processors increases, and that this is one of the reasons why superlinear speedups are obtained.

Semidirect parallel self-consistent field: the load balancing problem in the input/output intensive self-consistent field iterations

The full capacity of contemporary parallel computers can, in the context of iterative ab initio procedures like, for example, self-consistent field (SCF) and multiconfigurational SCF, only be utilized if the disk and input/output (I/O) capacity are fully exploited before the implementation turns to an integral direct strategy. In a recent report on parallel semidirect SCF http://www.tc.cornell.edu/er/media/1996/collabrate.html, http://www.fp.mcs.anl.gd/grand-challenges/chem/nondirect/index.html it was demonstrated that super-linear speedups are achievable for algorithms that exploit scalable parallel I/O. In the I/O-intensive SCF iterations of this implementation a static load balancing, however, was employed, dictated by the initial iteration in which integral evaluation dominates the central processing unit activity and thus determines the load balancing. In the present paper we present the first implementation in which load balancing is achieved throughout the whole SCF procedure, i.e. also in subsequent iterations. The improved scalability of our new algorithm is demonstrated in some test calculations, for example, for 63-node calculation a speedup of 104 was observed in the computation of the two-electron integral contribution to the Fock matrix.

Parallel algorithms for Bayesian phylogenetic inference

The combination of a Markov chain Monte Carlo (MCMC) method with likelihood-based assessment of phylogenies is becoming a popular alternative to direct likelihood optimization. However, MCMC, like maximum likelihood, is a computationally expensive method. To approximate the posterior distribution of phylogenies, a Markov chain is constructed, using the Metropolis algorithm, such that the chain has the posterior distribution of the parameters of phylogenies as its stationary distribution. This paper describes parallel algorithms and their MPI-based parallel implementation for MCMC-based Bayesian phylogenetic inference. Bayesian phylogenetic inference is computationally expensive both in time and in memory requirements. Our variations on MCMC and their implementation were done to permit the study of large phylogenetic problems. In our approach, we can distribute either entire chains or parts of a chain to different processors, since in current models the columns of the data are independent. Evaluations on a 32-node Beowulf cluster suggest the problem scales well. A number of important points are identified, including a superlinear speedup due to more effective cache usage and the point at which additional processors slow down the process due to communication overhead.

Linear scaling computation of the Fock matrix. VI. Data parallel computation of the exchange-correlation matrix

Recently, early onset linear scaling computation of the exchange-correlation matrix has been achieved using hierarchical cubature [J. Chem. Phys. 113, 10037 (2000)]. Hierarchical cubature differs from other methods in that the integration grid is adaptive and purely Cartesian, which allows for a straightforward domain decomposition in parallel computations; the volume enclosing the entire grid may be simply divided into a number of nonoverlapping boxes. In our data parallel approach, each box requires only a fraction of the total density to perform the necessary numerical integrations due to the finite extent of Gaussian-orbital basis sets. This inherent data locality may be exploited to reduce communications between processors as well as to avoid memory and copy overheads associated with data replication. Although the hierarchical cubature grid is Cartesian, naive boxing leads to irregular work loads due to strong spatial variations of the grid and the electron density. In this paper we describe equal time partitioning, which employs time measurement of the smallest sub-volumes (corresponding to the primitive cubature rule) to load balance grid-work for the next self-consistent-field iteration. After start-up from a heuristic center of mass partitioning, equal time partitioning exploits smooth variation of the density and grid between iterations to achieve load balance. With the 3-21G basis set and a medium quality grid, equal time partitioning applied to taxol (62 heavy atoms) attained a speedup of 61 out of 64 processors, while for a 110 molecule water cluster at standard density it achieved a speedup of 113 out of 128. The efficiency of equal time partitioning applied to hierarchical cubature improves as the grid work per processor increases. With a fine grid and the 6-311G(df,p) basis set, calculations on the 26 atom molecule α-pinene achieved a parallel efficiency better than 99% with 64 processors. For more coarse grained calculations, superlinear speedups are found to result from reduced computational complexity associated with data parallelism.