Prefix Sum Research Articles

Scalable GPU graph traversal

Breadth-first search (BFS) is a core primitive for graph traversal and a basis for many higher-level graph analysis algorithms. It is also representative of a class of parallel computations whose memory accesses and work distribution are both irregular and data-dependent. Recent work has demonstrated the plausibility of GPU sparse graph traversal, but has tended to focus on asymptotically inefficient algorithms that perform poorly on graphs with non-trivial diameter. We present a BFS parallelization focused on fine-grained task management constructed from efficient prefix sum that achieves an asymptotically optimal O (| V |+| E |) work complexity. Our implementation delivers excellent performance on diverse graphs, achieving traversal rates in excess of 3.3 billion and 8.3 billion traversed edges per second using single and quad-GPU configurations, respectively. This level of performance is several times faster than state-of-the-art implementations both CPU and GPU platforms.

Flexible Management on BSP Process Rescheduling: Offering Migration at Middleware and Application Levels

This article describes the rationales for developing jMigBSP - a Java programming library that offers object rescheduling. It was designed to work on grid computing environments and offers an interface that follows the BSP (Bulk Synchronous Parallel) style. jMigBSP’s main contribution focuses on the rescheduling facility in two different ways: (i) by using migration directives on the application coded irectly and (ii) through automatic load balancing at middleware level. Especially, this second idea is feasible thanks to the Java’s inheritance feature, in which transforms a simple jMigBSP application in amigratable one only by changing a single line of code. In addition, the presented library makes the object interaction easier by providing one-sided message passing directives and hides network latency through asynchronous communications. Finally, we developed three BSP applications: (i) Prefix Sum; (ii) Fractal Image Compression (FIC) and; (iii) Fast Fourier Transform (FFT).They show our library as viable solution to offer load balancing on BSP applications. Specially, the FIC results present gains up to 37% when applying migration directives inside the code. Finally, the FFT tests emphasize strength of jMigBSP. In this situation, it outperforms a native library denoted BSPlib when migration facilities take place. Keywords: Bulk Synchronous Parallel, rescheduling, Java, adaptation, object migration, grid computing.

Energy cost evaluation of parallel algorithms for multiprocessor systems

With the continuous development of hardware and software, Graphics Processor Units (GPUs) have been used in the general-purpose computation field. They have emerged as a computational accelerator that dramatically reduces the application execution time with CPUs. To achieve high computing performance, a GPU typically includes hundreds of computing units. The high density of computing resource on a chip brings in high power consumption. Therefore power consumption has become one of the most important problems for the development of GPUs. This paper analyzes the energy consumption of parallel algorithms executed in GPUs and provides a method to evaluate the energy scalability for parallel algorithms. Then the parallel prefix sum is analyzed to illustrate the method for the energy conservation, and the energy scalability is experimentally evaluated using Sparse Matrix-Vector Multiply (SpMV). The results show that the optimal number of blocks, memory choice and task scheduling are the important keys to balance the performance and the energy consumption of GPUs.

Fast and Simultaneous Data Aggregation Over Multiple Regions in Wireless Sensor Networks

As the applications of wireless sensor networks continue to expand, it is important to support fast and simultaneous data aggregation over multiple regions for advanced data analysis. In this paper, we propose a solution by using a novel distributed data structure called distributed data cube (DDC). A DDC maintains a set of special forms of aggregate values (prefix sum, prefix average, prefix max, and prefix min) in distributed sensor nodes. We will first present fast algorithms to build a DDC within a sharp time bound. Then, we will present efficient distributed query-processing algorithms to handle aggregate queries by using a DDC. For a query region with <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</i> sensor nodes, our algorithms can return within <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">O</i> (√ <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</i> ) time. Finally, extensive simulation studies confirm that a DDC can be built very quickly, which is consistent with the theoretical time bound. The network traffic injected while constructing a DDC is acceptable and also scalable as the network size grows. Query processing on a DDC is fast and energy efficient in terms of the time units needed and the number of messages incurred.

Movie-based representation of reduction operations in numerical computing

Movie-based visual programming is an approach for creating computational programs utilizing animation frames of space structure scanning correlated with arithmetic and logical operations. The assignment of the operations on the frame structure nodes is implemented by coloring corresponding structure areas and specifying appropriate formulas on those areas. This programming is based on special acquisition of data/knowledge to generate executable code and to help application programmers in decision making. In this paper, an approach is considered for representing group reduction procedures (reduction and prefix sums, products, maximum/minimum search, etc.) that are widely used in various applications and libraries oriented to parallel and distributed computations. Within a process, new visual constructs and symbols including their syntax and semantics are introduced to define the procedures. In addition, a method of automatic production of template programs supporting the code generation is presented. The applicability of the procedures for high-level visual programming is demonstrated by an algorithm implementing the least-squares fitting method.

Efficient Updates for OLAP Range Queries on Flash Memory

This paper explores efficient ways to use flash memory to store online analytical processing (OLAP) data. The particular type of queries considered are range queries using the aggregate functions SUM, COUNT and AVG. The asymmetric cost of reads and writes for flash memory gives higher importance to how updates are handled in a flash memory environment. A popular data structure used for answering OLAP range-sum queries is the prefix sum cube. It allows the range-sum query to be answered in constant time. However, updating the prefix sum cube is very expensive. To overcome this, the Δ-tree was proposed by Chun et al. (Dynamic update cube for range-sum queries. Proc. Int. Conf. Very Large Data Bases, San Francisco, CA, USA, 2001, pp. 521–530. Morgan Kaufmann Publisher). The Δ-tree stores all updates to the prefix sum cube in a separate r-tree. This approach worked well for the hard disk where in-place updates are relatively cheap. However, for flash memory where in-place updates are very expensive, the Δ-tree performs very poorly. We take a four-pronged approach to overcome the problem of expensive in-place updates. The first is efficient caching of updates in RAM. The second is writing out whole trees from RAM to flash memory instead of incrementally updating a disk resident tree. The third is we allow users to trade bounded amounts of accuracy for less updates via lossy compression. Finally, we use a quadtree index structure instead of the R-tree. We prove that the quadtree compression problem is NP-complete. A greedy heuristic is proposed to find near optimal solutions in polynomial time. Various experiments were conducted to compare the proposed algorithms against the existing Δ-tree. The results show that our algorithms consistently outperformed Δ-tree by factors of between 10 and 100. This demonstrates the importance of designing flash memory customized algorithms for OLAP range queries. In addition, among our algorithms, the error bound solutions with a small error bound setting significantly outperform the accurate solution in terms of performance for a variety of parameter settings. This indicates that the error bound algorithms offer users an effective trade-off between execution time and accuracy.

TU‐C‐BRA‐09: High‐Performance Dose Calculation for Proton Radiotherapy Using GPU

Purpose: To investigate the feasibility and the potential speed gain of GPU accelerated dose calculation for proton pencil beam algorithms with heterogeneity correction and to offer solutions to obstacles in implementing our algorithm. Method and Materials: We implemented our in‐house proton dose calculation system on an NVIDIA GTX 280 graphic card and an Intel Xeon 2.83 GHz processor using the Compute Unified Device Architecture environment. Several key techniques and strategies were employed in order to optimize the performance of the GPU code. (1) Modified the scaling algorithm used to calculate lateral spreading of the proton beam in the presence of tissue inhomogeneities. (2) Used an incremental ray tracing algorithm to reduce the memory requirement for each ray allowing for better parallelization. (3) The penalty of massive non‐coalesced memory reads required for convolution/superposition was alleviated by texture fetching. (4) Parallel algorithms such as reduction and prefix sum were implemented on GPU to help avoid data transfer between the CPU and GPU memory. The performance of both implementations was evaluated on a prostate clinical case. Results: With no loss in accuracy, the dose calculation time per beamlet with our GPU implementation ranges from 200 – 500 ms compared to 15 – 40 s on CPU. Approximately 80 times speed gain is achieved for various number of convolution/superposition steps. Conclusions: GPU‐based proton dose calculation is feasible with adapted algorithms and proper implementation techniques. Close to two magnitude of performance gain can be achieved with typical hardware. We believe GPU‐based fast dose calculation can reduce the routine treatment planning workload as well as provide a feasible solution to the realization of online adaptive intensity modulated proton therapy.

Simultaneous aggregate sum retrieval from multiple regions in sensor networks by distributed data cubes

Several data aggregation algorithms aiming at the retrieval of data aggregates from a single set of sensors in a sensor network have been proposed. However, when data aggregates of several sets of sensors are needed, the only solution these techniques provide is to build multiple distributed data structures or gossip groups in the sensor network. Hence in a sensor network containing N sensors, we may need 2 N distributed data structures or gossip groups in order to retrieve the aggregates from all possible sets of sensors. In this paper, we propose to build distributed data cubes for the fast retrieval of aggregate sums from multiple regions in a sensor network, such that only one distributed data structure is needed. The distributed data cube construction algorithms we propose are based on the inclusion-exclusion principle, and they are capable of building distributed Prefix Sum (PS) and Local Prefix Sum (LPS) data cubes in sensor networks.

Two-tree algorithms for full bandwidth broadcast, reduction and scan

We present a new, simple algorithmic idea for the collective communication operations broadcast, reduction, and scan (prefix sums). The algorithms concurrently communicate over two binary trees which both span the entire network. By careful layout and communication scheduling, each tree communicates as efficiently as a single tree with exclusive use of the network. Our algorithms thus achieve up to twice the bandwidth of most previous algorithms. In particular, our approach beats all previous algorithms for reduction and scan. Experiments on clusters with Myrinet and InfiniBand interconnect show significant reductions in running time for all three operations sometimes even close to the best possible factor of two.

A Formalization of Powerlist Algebra in ACL2

In Misra (ACM Trans Program Lang Syst 16(6):1737---1767, 1994), Misra introduced the powerlist data structure, which is well suited to express recursive, data-parallel algorithms. Moreover, Misra and other researchers have shown how powerlists can be used to prove the correctness of several algorithms. This success has encouraged some researchers to pursue automated proofs of theorems about powerlists (Kapur 1997; Kapur and Subramaniam 1995, Form Methods Syst Des 13(2):127---158, 1998). In this paper, we show how ACL2 can be used to verify theorems about powerlists. We depart from previous approaches in two significant ways. First, the powerlists we use are not the regular structures defined by Misra; that is, we do not require powerlists to be balanced trees. As we will see, this complicates some of the proofs, but on the other hand it allows us to state theorems that are otherwise beyond the language of powerlists. Second, we wish to prove the correctness of powerlist algorithms as much as possible within the logic of powerlists. Previous approaches have relied on intermediate lemmas which are unproven (indeed unstated) within the powerlist logic. However, we believe these lemmas must be formalized if the final theorems are to be used as a foundation for subsequent work, e.g., in the verification of system libraries. In our experience, some of these unproven lemmas presented the biggest obstacle to finding an automated proof. We illustrate our approach with two case studies involving Batcher sorting and prefix sums.

Permutation algorithms on optical multi-trees

The Optical Multi-Trees (OMULT) is an interconnection network proposed by Sinha and Bandyopadhyay [B.P. Sinha, S. Bandyopadhyay, OMULT: An optical interconnection system for parallel computing, Lecture notes in Computer Science 3149 (2004) 302–312], for optoelectronic parallel computers. Various algorithms including matrix multiplication, DFT computation, sorting, prefix sum have been successfully mapped on this architecture. In this paper, we develop efficient parallel algorithms for some commonly used permutations namely, bit reversal, vector reversal, perfect shuffle, unshuffle and transpose on the OMULT network. Our algorithm for bit reversal permutation requires 8logn electronic moves +7 optical moves for n2 data elements and O(n) electronic moves +3 optical moves for n3 data elements; the vector reversal for n3 data elements requires 3g(n) electronic moves +4 optical moves, where g(n) is the time for local vector reversal on n data elements; the perfect shuffle for n3 data elements requires (3f(n)+8) electronic moves +8 optical moves, where f(n) is the time for local perfect shuffle on n data elements, and the transpose for n3 data elements runs in at most three optical moves, all using 2n3–n2 processors.

Succinct indexable dictionaries with applications to encoding k -ary trees, prefix sums and multisets

We consider the indexable dictionary problem, which consists of storing a set S ⊆ {0,…, m − 1} for some integer m while supporting the operations of rank( x ), which returns the number of elements in S that are less than x if x ∈ S , and −1 otherwise; and select( i ), which returns the i th smallest element in S . We give a data structure that supports both operations in O (1) time on the RAM model and requires B( n, m ) + o ( n ) + O (lg lg m ) bits to store a set of size n , where B( n, m ) = ⌊lg ( m / n )⌋ is the minimum number of bits required to store any n -element subset from a universe of size m . Previous dictionaries taking this space only supported (yes/no) membership queries in O (1) time. In the cell probe model we can remove the O (lg lg m ) additive term in the space bound, answering a question raised by Fich and Miltersen [1995] and Pagh [2001]. We present extensions and applications of our indexable dictionary data structure, including: —an information-theoretically optimal representation of a k -ary cardinal tree that supports standard operations in constant time; —a representation of a multiset of size n from {0,…, m − 1} in B( n, m + n ) + o ( n ) bits that supports (appropriate generalizations of) rank and select operations in constant time; and + O (lg lg m ) —a representation of a sequence of n nonnegative integers summing up to m in B( n, m + n ) + o ( n ) bits that supports prefix sum queries in constant time.

Automatic inversion generates divide-and-conquer parallel programs

Divide-and-conquer algorithms are suitable for modern parallel machines, tending to have large amounts of inherent parallelism and working well with caches and deep memory hierarchies. Among others, list homomorphisms are a class of recursive functions on lists, which match very well with the divide-and-conquer paradigm. However, direct programming with list homomorphisms is a challenge for many programmers. In this paper, we propose and implement a novel systemthat can automatically derive cost-optimal list homomorphisms from a pair of sequential programs, based on the third homomorphism theorem. Our idea is to reduce extraction of list homomorphisms to derivation of weak right inverses. We show that a weak right inverse always exists and can be automatically generated from a wide class of sequential programs. We demonstrate our system with several nontrivial examples, including the maximum prefix sum problem, the prefix sum computation, the maximum segment sum problem, and the line-of-sight problem. The experimental results show practical efficiency of our automatic parallelization algorithm and good speedups of the generated parallel programs.

The cell probe complexity of succinct data structures

We consider time-space tradeoffs for static data structure problems in the cell probe model with word size 1 (the bit probe model). In this model, the goal is to represent n -bit data with s = n + r bits such that queries (of a certain type) about the data can be answered by reading at most t bits of the representation. Ideally, we would like to keep both s and t small, but there are tradeoffs between the values of s and t that limit the possibilities of keeping both parameters small. In this paper, we consider the case of succinct representations, where s = n + r for some redundancy r ≪ n . For a Boolean version of the problem of polynomial evaluation with preprocessing of coefficients, we show a lower bound on the redundancy–query time tradeoff of the form ( r + 1 ) t ≥ Ω ( n / log n ) . In particular, for very small redundancies r , we get an almost optimal lower bound stating that the query algorithm has to inspect almost the entire data structure (up to a logarithmic factor). We show similar lower bounds for problems satisfying a certain combinatorial properties of a coding theoretic flavor, and obtain ( r + 1 ) t ≥ Ω ( n ) for certain problems. Previously, no ω ( m ) lower bounds were known on t in the general model for explicit Boolean problems, even for very small redundancies. By restricting our attention to systematic or index structures ϕ satisfying ϕ ( x ) = x ⋅ ϕ ∗ ( x ) for some map ϕ ∗ (where ⋅ denotes concatenation), we show similar lower bounds on the redundancy–query time tradeoff for the natural data structuring problems of Prefix Sum and Substring Search.

O(log*n) algorithms on a Sum-CRCW PRAM

We present work- and cost-optimal O(log*n) algorithms for prefix sums and linear integer sorting on a Sum-CRCW PRAM.

On a class of cell circuits

We introduce a class of cell circuits, T -circuits, and describe a connection between the lower bound for the area and the depth of the circuits of this class: the less the depth the greater the area of a circuit. We give examples of T -circuits with logarithmic depth in the problem of calculation of n prefix sums and also of sums and differences of two n -digit numbers. It is shown that the area of these circuits is O ( n log n ) and has the optimal order.

Hypercube computations on partitioned optical passive stars networks

This paper shows that an n=2k processor partitioned optical passive stars (POPS) network with g groups and d processors per group can simulate every bidirectional move of an n processor hypercube using one slot when d g. Moreover, the same POPS network can simulate every monodirectional move of a processor hypercube using one slot when d=g. All these results are shown to be optimal. Our simulations improve on the literature whenever dneg and directly yield several important consequences. For example, as a direct consequence of our simulations, a POPS network, n=dg and d<g, can compute the prefix sums of n data values in log2n slots. This is faster than the best previously known ad hoc algorithm and is actually optimal. Similarly, we improve on the best POPS network algorithms for both the prefix sums problem on general POPS networks and the fundamental online permutation routing problem, among others

On a class of cell circuits

We introduce a class of cell circuits, T-circuits, and describe a connection between the lower bound for the area and the depth of the circuits of this class: the less the depth the greater the area of a circuit. We give examples of T-circuits with logarithmic depth in the problem of calculation of n prefix sums and also of sums and differences of two n-digit numbers. It is shown that the area of these circuits is O(n log n) and has the optimal order.

CGMGRAPH/CGMLIB: Implementing and Testing CGM Graph Algorithms on PC Clusters and Shared Memory Machines

In this paper, we present CGMgraph, the first integrated library of parallel graph methods for PC clusters based on Coarse Grained Multicomputer (CGM) algorithms. CGMgraph implements parallel methods for various graph problems. Our implementations of deterministic list ranking, Euler tour, connected components, spanning forest, and bipartite graph detection are, to our knowledge, the first efficient implementations for PC clusters. Our library also includes CGMlib, a library of basic CGM tools such as sorting, prefix sum, one-to-all broadcast, all-to-one gather, h-Relation, all-to-all broadcast, array balancing, and CGM partitioning. Both libraries are available for download at http://www.scs.carleton.ca/~cgm. In the experimental part of this paper, we demonstrate the performance of our methods on four different architectures: a gigabit connected high performance PC cluster, a smaller PC cluster connected via fast ethernet, a network of workstations, and a shared memory machine. Our experiments show that our library provides good parallel speedup and scalability on all four platforms. The communication overhead is, in most cases, small and does not grow significantly with an increasing number of processors. This is a very important feature of CGM algorithms which makes them very efficient in practice.

Constant time fault tolerant algorithms for a linear array with a reconfigurable pipelined bus system

Recently, researchers have proposed many models using reconfigurable optically pipelined buses. Most algorithms developed for these models assume that a healthy system is available. The only previous work in this area considers faulty processors or hardware for an N-processor linear array with a reconfigurable pipelined bus system (LARPBS), resulting in fault tolerant algorithms that run in O ( log N ) time. We extend and improve upon previous results by considering new fault scenarios and slight modifications to the LARPBS model. By modifying the switches on the receiving segment of an LARPBS, a reasonable, low-cost extension to the model, we are able to achieve fault tolerant algorithms that execute in O ( 1 ) time rather than O ( log N ) time. The fault tolerant algorithms that we consider are key basic fundamental algorithms, such as compaction, binary prefix sums, and sorting, that all execute in constant time on a healthy LARPBS. As the algorithms improved are building blocks for more complex algorithms, the results presented in this paper will extend to those more complex algorithms as well.