Prefix Sum Research Articles

Multi-Function Scan Circuit for Assisting the Parallel Computational Map Pattern

Some parallel computing patterns can be accelerated using appropriate networks of simple circuits. We propose a solution, based on the Beneš-Waksman permutation network, which is adapted to efficiently accelerate not only permutation, but some of the most used parallel computational patterns such as: pack, prefix operations and reductions. The structural context considered for deploying our circuit is the map parallel pattern represented by an array of computational elements. The developed network receives a vector from the map array and outputs a vector for functions such as permute, pack, prefix sum (thus closing a first global loop over the map array of cells). For reduction functions (add, min, max) the network returns a scalar (thus closing a second global loop over the map array). With these improvements, this network adds circuit support for frequently used functions, in addition to map-type functions performed in the array of computing elements. While for reduction functions the frame of the permutation network can be easily adapted, for prefix functions and for the pack function new forms of implementation are proposed. The cells of the Bene\v s-Waksman network are redesigned to support the additional functionality. Some applications are then presented to emphasize the utility of our design.

Totally-ordered Sequential Rules for Utility Maximization

High-utility sequential pattern mining (HUSPM) is a significant and valuable activity in knowledge discovery and data analytics with many real-world applications. In some cases, HUSPM can not provide an excellent measure to predict what will happen. High-utility sequential rule mining (HUSRM) discovers high utility and high confidence sequential rules, so it can solve the issue in HUSPM. However, all existing HUSRM algorithms aim to find high-utility partially-ordered sequential rules (HUSRs), which are not consistent with reality and may generate fake HUSRs. Therefore, in this article, we formulate the problem of high-utility totally-ordered sequential rule mining and propose a novel algorithm, called TotalSR, which aims to identify all high-utility totally-ordered sequential rules (HTSRs). TotalSR introduces a left-first expansion strategy that can utilize the anti-monotonic property to use a confidence pruning strategy. TotalSR also designs a new utility upper bound: RSPEU , which is tighter than the existing upper bounds. TotalSR can drastically reduce the search space with the help of utility upper bounds pruning strategies, avoiding much more meaningless computation. To effectively compute the information, TotalSR proposes an auxiliary antecedent record table that can efficiently calculate the antecedent’s support and a utility prefix sum list that can compute the upper bound in O (1) time for a sequence. Finally, there are numerous experimental results on both real and synthetic datasets demonstrating that TotalSR is more efficient than the existing algorithms.

Smoother: on-the-fly processing of interactome data using prefix sums.

Nucleic acid interactome data, such as chromosome conformation capture data and RNA-DNA interactome data, are currently analyzed via pipelines that must be rerun for each new parameter set. A more dynamic approach is desirable since the optimal parameter set is commonly unknown ahead of time and rerunning pipelines is a time-consuming process. We have developed an approach fast enough to process interactome data on-the-fly using a sparse prefix sum index. With this index, we created Smoother, a flexible, multifeatured visualization and analysis tool that allows interactive filtering, e.g. by mapping quality, almost instant comparisons between different normalization approaches, e.g. iterative correction, and ploidy correction. Further, Smoother can overlay other sequencing data or genomic annotations, compare different samples, and perform virtual 4C analysis. Smoother permits a novel way to interact with and explore interactome data, fostering comprehensive, high-quality data analysis. Smoother is available at https://github.com/Siegel-Lab/BioSmoother under the MIT license.

A portable C++ library for memory and compute abstraction on multi‐core CPUs and GPUs

AbstractWe present a C++ library for transparent memory and compute abstraction across CPU and GPU architectures. Our library combines generic data structures like vectors, multi‐dimensional arrays, maps, graphs, and sparse grids with basic generic algorithms like arbitrary‐dimensional convolutions, copying, merging, sorting, prefix sum, reductions, neighbor search, and filtering. The memory layout of the data structures is adapted at compile time using C++ tuples with optional memory double‐mapping between host and device and the capability of using memory managed by external libraries with no data copying. We combine this transparent memory layout with generic thread‐parallel algorithms under two alternative common interfaces: a CUDA‐like kernel interface and a lambda‐function interface. We quantify the memory and compute performance and portability of our implementation using micro‐benchmarks, showing that the abstractions introduce negligible performance overhead, and we compare performance against the current state of the art in a real‐world scientific application from computational fluid mechanics.

When does 0–1 Principle Hold for Prefix Sums?

Knuth’s 0–1 principle argues that the correctness of any swap-based sorting network can be verified by testing arbitrary sequences over Boolean values (i.e., 0 and 1). Voigtländer (Proceedings of the 35th ACM SIGPLAN-SIGACT symposium on principles of programming languages, POPL 2008, San Francisco, California, USA, January 7–12, 2008. ACM, New York, NY, pp 29–35, 2008. https://doi.org/10.1145/1328438.1328445) proved a similar result for prefix-sum networks that consist of associative binary operators: the correctness can be verified by testing arbitrary sequences and associative binary operators over three values, namely 0, 1, and 2. He raised the question of whether testing over Boolean values is sufficient if the binary operator is idempotent in addition to associative. This paper answers his question. First, there is an incorrect prefix-sum network for associative idempotent operators, the flaw of which cannot be detected by testing over Boolean values. Second, testing over Boolean values is sufficient if the binary operators are restricted to commutative in addition to associative and idempotent.

Elementary Algorithms – Prefix Sum

In this paper we present a special problem, named as prefix sum, and its variations. We analyse the base problem more detailed and compare the possible solutions. Our students solved simi-lar problems in competitions, so we have statistics about their solutions. We offer this article for those readers who are interested in programming methodology, or who are teachers, and their students will be participated in competitions.

An Improved Parallel Prefix Sums Algorithm

The prefix sums problem for an array [Formula: see text] is to compute all sums [Formula: see text], [Formula: see text]. In this paper, we introduce an improvement for the best previous algorithm that runs in [Formula: see text] time using [Formula: see text] processors on a Sum Concurrent Read Concurrent Write, Parallel Random Access Machine (Sum-CRCW PRAM). The improvements include (1) reducing the total number of operations and (2) reducing the amount of storage required by the algorithm.

Formal verification of parallel prefix sum and stream compaction algorithms in CUDA

GPUs are an important part of any High Performance Computing (HPC) architecture. To make optimal use of the specifics of a GPU architecture, we need programming models that naturally support the parallel execution model of a GPU. CUDA and OpenCL are two widely used examples of such programming models. Furthermore, we also need to redesign algorithms such that they adhere to this parallel programming model, and we need to be able to prove the correctness of these redesigned algorithms.In this paper we study two examples of such parallelized algorithms, and we discuss how to prove their correctness (data race freedom and (partial) functional correctness) using the VerCors program verifier. First of all, we prove the correctness of two parallel algorithms solving the prefix sum problem. Second, we show how such a prefix sum algorithm is used as a basic block in a stream compaction algorithm, and we prove correctness of this stream compaction algorithm, taking advantage of the earlier correctness proof for the prefix sum algorithm.The proofs as described in this paper are developed over the CUDA implementations of these algorithms. In earlier work, we had already shown correctness of a more high-level version of the algorithm. This paper discusses how we add support to reason about CUDA programs in VerCors, and it then shows how we can redo the verification at the level of the CUDA code. We also discuss some practical challenges that we had to address to prove correctness of the actual CUDA-level verifications.

Parallel Makespan Calculation for Flow Shop Scheduling Problem with Minimal and Maximal Idle Time

In this paper, a flow shop scheduling problem with minimal and maximal machine idle time with the goal of minimizing makespan is considered. The mathematical model of the problem is presented. A generalization of the prefix sum, called the job shift scan, for computing required shifts for overlapping jobs is proposed. A work-efficient algorithm for computing the job shift scan in parallel for the PRAM model with n processors is proposed and its time complexity of O(logn) is proven. Then, an algorithm for computing the makespan in time O(mlogn) in parallel using the prefix sum and job shift scan is proposed. Computer experiments on GPU were conducted using the CUDA platform. The results indicate multi-thread GPU vs. single-thread GPU speedups of up to 350 and 1000 for job shift scan and makespan calculation algorithms, respectively. Multi-thread GPU vs. single-thread CPU speedups up to 4.5 and 14.7, respectively, were observed as well. The experiments on the Taillard-based problem instances using a simulated annealing solving method and employing the parallel makespan calculation show that the method is able to perform many more iterations in the given time limit and obtain better results than the non-parallel version.

Parameterized Splitting of Summed Volume Tables

AbstractSummed Volume Tables (SVTs) allow one to compute integrals over the data values in any cubical area of a three‐dimensional orthogonal grid in constant time, and they are especially interesting for building spatial search structures for sparse volumes. However, SVTs become extremely memory consuming due to the large values they need to store; for a dataset of n values an SVT requires 𝒪(n log n) bits. The 3D Fenwick tree allows recovering the integral values in 𝒪(log3 n) time, at a memory consumption of 𝒪(n) bits. We propose an algorithm that generates SVT representations that can flexibly trade speed for memory: From similar characteristics as SVTs, over equal memory consumption as 3D Fenwick trees at significantly lower computational complexity, to even further reduced memory consumption at the cost of raising computational complexity. For a 641 × 9601 × 9601 binary dataset, the algorithm can generate an SVT representation that requires 27.0GB and 46 · 8 data fetch operations to retrieve an integral value, compared to 27.5GB and 1521·8 fetches by 3D Fenwick trees, a decrease in fetches of 97%. A full SVT requires 247.6GB and 8 fetches per integral value. We present a novel hierarchical approach to compute and store intermediate prefix sums of SVTs, so that any prescribed memory consumption between 𝒪(n) bits and 𝒪(n log n) bits is achieved. We evaluate the performance of the proposed algorithm in a number of examples considering large volume data, and we perform comparisons to existing alternatives.

Simplifying dependent reductions in the polyhedral model

A Reduction – an accumulation over a set of values, using an associative and commutative operator – is a common computation in many numerical computations, including scientific computations, machine learning, computer vision, and financial analytics. Contemporary polyhedral-based compilation techniques make it possible to optimize reductions, such as prefix sums, in which each component of the reduction’s output potentially shares computation with another component in the reduction. Therefore an optimizing compiler can identify the computation shared between multiple components and generate code that computes the shared computation only once. These techniques, however, do not support reductions that – when phrased in the language of the polyhedral model – span multiple dependent statements. In such cases, existing approaches can generate incorrect code that violates the data dependences of the original, unoptimized program. In this work, we identify and formalize the optimization of dependent reductions as an integer bilinear program. We present a heuristic optimization algorithm that uses an affine sequential schedule of the program to determine how to simplfy reductions yet still preserve the program’s dependences. We demonstrate that the algorithm provides optimal complexity for a set of benchmark programs from the literature on probabilistic inference algorithms, whose performance critically relies on simplifying these reductions. The complexities for 10 of the 11 programs improve siginifcantly by factors at least of the sizes of the input data, which are in the range of 10 4 to 10 6 for typical real application inputs. We also confirm the significance of the improvement by showing speedups in wall-clock time that range from 1.1x to over 10 6 x.

Parallel Differential Evolutionary Particle Filtering Algorithm Based on the CUDA Unfolding Cycle

Aiming at the problem of low statute efficiency of prefix sum execution during the execution of the parallel differential evolutionary particle filtering algorithm, a filtering algorithm based on the CUDA unfolding cyclic prefix sum is proposed to remove the thread differentiation and thread idleness existing in the parallel prefix sum by unfolding the cyclic method and unfolding the thread bundle method, optimize the cycle, and improve the prefix sum execution efficiency. By introducing the parallel strategy, the differential evolutionary particle filtering algorithm is implemented in parallel and executed on the GPU side using the improved prefix sum computation during the algorithm update. Through big data analysis, the results show that this parallel differential evolutionary particle filtering algorithm with the improved prefix sum statute can effectively improve differential evolutionary particle filtering for nonlinear system states and real‐time performance in heterogeneous parallel processing systems.

Автоматизация разработки на Vulkan: предметно-ориентированный подход

In this paper we propose a high-level approach to developing GPU applications based on the Vulkan API. The purpose of the work is to reduce the complexity of developing and debugging applications that implement complex algorithms on the GPU using Vulkan. The proposed approach uses the technology of code generation by translating a C++ program into an optimized implementation in Vulkan, which includes automatic shader generation, resource binding, and the use of synchronization mechanisms (Vulkan barriers). The proposed solution is not a general-purpose programming technology, but specializes in specific tasks. At the same time, it has extensibility, which allows to adapt the solution to new problems. For single input C++ program, we can generate several implementations for different cases (via translator options) or different hardware. For example, a call to virtual functions can be implemented either through a switch construct in a kernel, or through sorting threads and an indirect dispatching via different kernels, or through the so-called callable shaders in Vulkan. Instead of creating a universal programming technology for building various software systems, we offer an extensible technology that can be customized for a specific class of applications. Unlike, for example, Halide, we do not use a domain-specific language, and the necessary knowledge is extracted from ordinary C++ code. Therefore, we do not extend with any new language constructs or directives and the input source code is assumed to be normal C++ source code (albeit with some restrictions) that can be compiled by any C++ compiler. We use pattern matching to find specific patterns (or patterns) in C++ code and convert them to GPU efficient code using Vulkan. Pattern are expressed through classes, member functions, and the relationship between them. Thus, the proposed technology makes it possible to ensure a cross-platform solution by generating different implementations of the same algorithm for different GPUs. At the same time, due to this, it allows you to provide access to specific hardware functionality required in computer graphics applications. Patterns are divided into architectural and algorithmic. The architectural pattern defines the domain and behavior of the translator as a whole (for example, image processing, ray tracing, neural networks, computational fluid dynamics and etc.). Algorithmic pattern express knowledge of data flow and control and define a narrower class of algorithms that can be efficiently implemented in hardware. Algorithmic patterns can occur within architectural patterns. For example, parallel reduction, compaction (parallel append), sorting, prefix sum, histogram calculation, map-reduce, etc. The proposed generator works on the principle of code morphing. The essence of this approach is that, having a certain class in the program and transformation rules, one can automatically generate another class with the desired properties (for example, the implementation of the algorithm on the GPU). The generated class inherits from the input class and thus has access to all data and functions of the input class. Overriding virtual functions in generated class helps user to carefully connect generated code to the other Vulkan code written by hand. Shaders can be generated in two variants: OpenCL shaders for google “clspv” compiler and GLSL shaders for an arbitrary GLSL compiler. Clspv variant is better for code which intensively uses pointers and the GLSL generator is better if specific HW features are used (like hardware ray tracing acceleration). We have demonstrated our technology on several examples related to image processing and ray tracing on which we get 30-100 times acceleration over multithreaded CPU implementation.

Lambda calculus with algebraic simplification for reduction parallelisation: Extended study

Abstract Parallel reduction is a major component of parallel programming and widely used for summarisation and aggregation. It is not well understood, however, what sorts of non-trivial summarisations can be implemented as parallel reductions. This paper develops a calculus named λAS, a simply typed lambda calculus with algebraic simplification. This calculus provides a foundation for studying a parallelisation of complex reductions by equational reasoning. Its key feature is δ abstraction. A δ abstraction is observationally equivalent to the standard λ abstraction, but its body is simplified before the arrival of its arguments using algebraic properties such as associativity and commutativity. In addition, the type system of λAS guarantees that simplifications due to δ abstractions do not lead to serious overheads. The usefulness of λAS is demonstrated on examples of developing complex parallel reductions, including those containing more than one reduction operator, loops with conditional jumps, prefix sum patterns and even tree manipulations.

Optimized Parallel Prefix Sum Algorithm on Optoelectronic Biswapped-Torus Architecture

The Biswapped-Torus is a recently reported optoelectronic node-symmetrical member of Biswapped-framework family. In this paper,optimized parallel approach is presented for prefix sum computation on [Formula: see text] Biswapped-Torus. The proposed parallel algorithm demands total 7[Formula: see text] electronic and three optical moves on odd network size or 7[Formula: see text] electronic and three optical moves on even network size. The algorithmic performance of the suggested parallel algorithm is also compared with the performances of recently reported optimal prefix sum algorithms on [Formula: see text] Biswapped-Mesh and [Formula: see text]-dimensional Biswapped Hyper Hexa-cell. Based on the comparative analysis, Biswapped-Torus claims to map prefix sum faster that require fewer communication moves compared to the Grid-based traditional architecture of biswapped family named Biswapped-Mesh. Moreover, the former also has architectural benefit of node-symmetry that leads to advantages such as easy embedding, mapping and designing of routing algorithms. Compared to symmetrical counter-part of biswapped family named Biswapped-Hyper Hexa-Cell, Biswapped-Torus is cost-efficient, but requires comparatively more communication moves for mapping prefix sum.

Optimal Parallel Prefix Sum Computation on Three-Dimensional Torus Network

In this article, an optimal parallel algorithm is suggested for fast computation of a crucial yet fundamental numerical problem in science named ‘prefix sum’ on a popular three-dimensional torus network. The proposed parallel algorithm demands total ‘4n + 2’ communication moves for mapping prefix sum problem on odd network sizes of n × n×n three-dimensional torus network, whereas total ‘4n’ communication moves is required for parallel mapping of same on even network sizes. In fact, a unit increase in each dimension from odd to even and even to odd network sizes resulted in the constant increase of 2 and 6 units, respectively, in the required communication moves for mapping prefix sum. The algorithmic performance of the proposed parallel algorithm can be compared to counter-part parallel prefix sum approaches on grid-based networks.

A completely parallel surface reconstruction method for particle-based fluids

We present a novel surface reconstruction pipeline that significantly improves reconstructing efficiency while preserves high-quality details for particle-based liquid simulation. Our surface reconstruction algorithm is a sort of completely parallel narrow band method. At the beginning of reconstruction, we develop a spatial hashing grid-based strategy to identify surface particles, which is much more precise and simpler than the smoothed color field. Consequently, those precise surface particles ensure accurate extraction of scalar field in the narrow band around surface without any redundancy, which brings great performance improvement for subsequent reconstruction stages. Furthermore, in order to obtain a better computation performance, we carefully analyze the potential race conditions and conditional branches of each reconstruction step between parallel threads and come up with a completely parallel reconstruction method combined with the exclusive prefix sum algorithm. Our method is pretty straightforward to implement. Experimental results demonstrate that our method runs up to dozen times faster than the state-of-the-art of narrow band-based fluid surface reconstruction, especially for large-scale particle-based fluid.

Compact Fenwick trees for dynamic ranking and selection

SummaryThe Fenwick tree is a classical implicit data structure that stores an array in such a way that modifying an element, accessing an element, computing a prefix sum and performing a predecessor search on prefix sums all take logarithmic time. We introduce a number of variants which improve the classical implementation of the tree: in particular, we can reduce its size when an upper bound on the array element is known, and we can perform much faster predecessor searches. Our aim is to use our variants to implement an efficient dynamic bit vector: our structure is able to perform updates, ranking and selection in logarithmic time, with a space overhead in the order of a few percents, outperforming existing data structures with the same purpose. Along the way, we highlight the pernicious interplay between the arithmetic behind the Fenwick tree and the structure of current CPU caches, suggesting simple solutions that improve performance significantly.

Differentially Private Real-time Streaming Data Publication Based on Sliding Window under Exponential Decay

Continuous response of range query on steaming data provides useful information for many practical applications as well as the risk of privacy disclosure. The existing research on differential privacy streaming data publication mostly pay close attention to boosting query accuracy, but pay less attention to query efficiency, and ignore the effect of timeliness on data weight. In this paper, we propose an effective algorithm of differential privacy streaming data publication under exponential decay mode. Firstly, by introducing the Fenwick tree to divide and reorganize data items in the stream, we achieve a constant time complexity for inserting a new item and getting the prefix sum. Meanwhile, we achieve time complicity linear to the number of data item for building a tree. After that, we use the advantage of matrix mechanism to deal with relevant queries and reduce the global sensitivity. In addition, we choose proper diagonal matrix further improve the range query accuracy. Finally, considering about exponential decay, every data item is weighted by the decay factor. By putting the Fenwick tree and matrix optimization together, we present complete algorithm for differentiate private real-time streaming data publication. The experiment is designed to compare the algorithm in this paper with similar algorithms for streaming data release in exponential decay. Experimental results show that the algorithm in this paper effectively improve the query efficiency while ensuring the quality of the query.

Sparse prefix sums: Constant-time range sum queries over sparse multidimensional data cubes

Prefix sums are a powerful technique to answer range-sum queries over multi-dimensional arrays in O(1) time by looking up a constant number of values in an array of size O(N), where N is the number of cells in the multi-dimensional array. However, the technique suffers from O(N) update and storage costs. Relative prefix sums address the high update costs by partitioning the array into blocks, thereby breaking the dependency between cells.In this paper, we present sparse prefix sums that exploit data sparsity to reduce the high storage costs of relative prefix sums. By building upon relative prefix sums, sparse prefix sums achieve the same update complexity as relative prefix sums. The authors of relative prefix sums erroneously claimed that the update complexity is O(N) for any number of dimensions. We show that this claim holds only for two dimensions, whereas the correct complexity for an arbitrary number of d dimensions is O(Nd−1d). To reduce the storage costs, the sparse prefix sums technique exploits sparsity in the data and avoids to materialize prefix sums for empty rows and columns in the data grid; instead, look-up tables are used to preserve constant query time. Sparse prefix sums are the first approach to achieve O(1) query time with sub-linear storage costs for range-sum queries over sparse low-dimensional arrays. A thorough experimental evaluation shows that the approach works very well in practice. On the tested real-world data sets the storage costs are reduced by an order of magnitude with only a small overhead in query time, thus preserving microsecond-fast query answering.