Scalable Parallel Programming Research Articles

A Tag-Based Solution for Safer Parallel STL Usage

Abstract The continuous evolution of programming languages creates a challenge not only for the compiler implementers but also for the regular developers. The C++17 standard introduced the Parallel STL library, a parallel extension of the original and fundamental Standard Template Library (STL) with the promise to achieve safe, efficient, and scalable parallel programming in an easy way. While the syntax remained familiar, the new parallel algorithms introduced additional requirements, which are often ignored even by experienced programmers. In this paper, we investigate the root cause of the misuse of the Parallel STL. We carried out a classroom experiment to measure the programmers’ behaviour when programming using the Parallel STL and found that the major issue is not that their code would be erroneous, but the lack of awareness of the semantic requirements of parallel functors. While a fully automatic detection of erroneous constructs is computationally infeasible to achieve, we introduce a framework with which the developer will be able to indicate – axiomatically, based on absolute trust – that an operation has “safe” properties required by the algorithms. We implemented a prototype of the proposed framework to demonstrate its usability and effectiveness.

A Microservice Approach for a Cellular Automata Parallel Programming Environment

This paper presents an architecture for a parallel programming environment designed to assist the process of writing parallel programs. The proposed architecture allows the integration of several tools such as parallel programming skeletons and parallelizing compilers, among others. On the one hand, parallel programming skeletons simplify the design of parallel algorithms. On the other hand, parallelizing compilers rely on compiler capabilities to translate sequential programs into ones capable of using multiple processing units (CPU/GPU) found in personal computers (PCs). The technological solution presented in this work follows a microservices-oriented approach. This approach eases the integration of new parallel programming tools in the existing deployment through well-known protocols and existing services. We show how the aforementioned parallel programming tools were integrated to support the coding and parallelization of a general purpose cellular automata. Since the parallelization rely on the capabilities underlying to each tool, the focus of this work is not to deal with an efficient cellular automata parallelization. Instead, a promising microservice approach to the development of an extensible and scalable parallel programming environment is presented.

Disentanglement in nested-parallel programs

Nested parallelism has proved to be a popular approach for programming the rapidly expanding range of multicore computers. It allows programmers to express parallelism at a high level and relies on a run-time system and a scheduler to deliver efficiency and scalability. As a result, many programming languages and extensions that support nested parallelism have been developed, including in C/C++, Java, Haskell, and ML. Yet, writing efficient and scalable nested parallel programs remains challenging, primarily due to difficult concurrency bugs arising from destructive updates or effects. For decades, researchers have argued that functional programming can simplify writing parallel programs by allowing more control over effects but functional programs continue to underperform in comparison to parallel programs written in lower-level languages. The fundamental difficulty with functional languages is that they have high demand for memory, and this demand only grows with parallelism. In this paper, we identify a memory property, called disentanglement, of nested-parallel programs, and propose memory management techniques for improved efficiency and scalability. Disentanglement allows for (destructive) effects as long as concurrently executing threads do not gain knowledge of the memory objects allocated by each other. We formally define disentanglement by considering an ML-like higher-order language with mutable references and presenting a dynamic semantics for it that enables reasoning about computation graphs of nested parallel programs. Based on this graph semantics, we formalize a classic correctness property---determinacy race freedom---and prove that it implies disentanglement. This establishes that disentanglement applies to a relatively broad class of parallel programs. We then propose memory management techniques for nested-parallel programs that take advantage of disentanglement for improved efficiency and scalability. We show that these techniques are practical by extending the MLton compiler for Standard ML to support this form of nested parallelism. Our empirical evaluation shows that our techniques are efficient and scale well.

Using relaxed concurrent data structures for contention minimization in multithreaded MPI programs

Parallel computing is one of the top priorities in computer science. The main means of parallel processing information is a distributed computing system (CS) - a composition of elementary machines that interact through a communication medium. Modern distributed VSs implement thread-level parallelism (TLP) within a single computing node (multi-core CS with shared memory), as well as process-level parallelism (PLP) process-level parallelism for the entire distributed CS. The main tool for developing parallel programs for such systems is the MPI standard. The need to create scalable parallel programs that effectively use compute nodes with shared memory has determined the development of the MPI standard, which today supports the creation of hybrid multi-threaded MPI programs. A hybrid multi-threaded MPI program is the combination of the computational capabilities of processes and threads. The standard defines four types of multithreading: Single - one thread of execution; Funneled - a multi-threaded program, but only main thread can perform MPI operations; Serialized - only one thread at the exact same time can make a call to MPI functions; Multiple - each program flow can perform MPI functions at any time. The main task of the multiple mode is the need to synchronize the communication flows within each process. This paper presents an overview of the work that addresses the problem of synchronizing processes running on remote machines and synchronizing internal program threads. Method for synchronization of threads based on queues with weakened semantics of operations is proposed.

RESOURCE-INDEPENDENT REPRESENTATION OF SORTING NETWORKS IN Set@l PROGRAMMING LANGUAGE

The development of scalable parallel programs for high-performance computer systems with reconfigurable and hybrid architectures is an important problem in the field of modern supercomputer engineering. Resource-independent applied software in the high-level COLAMO programming language describes only the limited number of the algorithm implementations for some configurations due to the inductive principle of the information graph forming. However, the creation of efficient resource-independent parallel software requires programming tool that supports both induction and reduction of calculations. To solve the problem, we propose the application of the Set@l programming language based on the aspect-oriented programming paradigm and set-theoretical code representation. According to the aspect-oriented concept, Set@l describes an algorithm for the problem solution and its parallelizing and other implementation features as separated program modules – source code and various aspects. The programming language being considered has much in common with traditional set-theory-based languages such as SETL, SETLX and SETL2, but Set@l introduces the essential classification of collections by parallelism, definiteness, orderliness and other criteria. In contrast to the high-level COLAMO programming language, Set@l specifies the full information graph of a computational problem in the absolutely parallel form and describes the reduction principles for all configurations of a computer system as independent aspects of a program. In this paper, we demonstrate that the aspect-oriented sorting program in the Set@l language defines different types of quadratic sorting networks and various methods of their partition and reduction.

Easy PRAM-Based High-Performance Parallel Programming with ICE

Parallel machines have become more widely used. Unfortunately parallel programming technologies have advanced at a much slower pace except for regular programs. For irregular programs, this advancement is inhibited by high synchronization costs, non-loop parallelism, non-array data structures, recursively expressed parallelism and parallelism that is too fine-grained to be exploitable. We present ICE, a new parallel programming language that is easy-to-program, since: (i) ICE is a synchronous, lock-step language so there is no need for programmer-specified synchronization; (ii) for a PRAM algorithm its ICE program amounts to directly transcribing it; and (iii) the PRAM algorithmic theory offers unique wealth of parallel algorithms and techniques. We propose ICE to be a part of an ecosystem consisting of the XMT architecture, the PRAM algorithmic model, and ICE itself, that together deliver on the twin goal of easy programming and efficient parallelization of irregular programs. The XMT architecture, developed at UMD, can exploit fine-grained parallelism in irregular programs. We have built the ICE compiler which translates the ICE language into the multithreaded XMTC language; the significance of this is that multi-threading is a feature shared by practically all current scalable parallel programming languages thus providing a method to compile ICE code. As one indication of ease of programming, we observed a reduction in code size in 11 out of 16 benchmarks as compared to hand-optimized XMTC. For these programs, the average reduction in number of lines of code was 35.5 percent. The remaining 5 benchmarks had almost the same code size for both ICE and hand-optimized XMTC. Our main result is perhaps surprising: The run-time was comparable to XMTC with a 0.53 percent average gain for ICE across all benchmarks.

Queue Delegation Locking

The scalability of parallel programs is often bounded by the performance of synchronization mechanisms used to protect critical sections. The performance of these mechanisms is in turn determined by their sequential execution time, efficient use of hardware, and ability to avoid waiting. In this article, we describe queue delegation (QD) locking , a family of locks that both delegate critical sections and enable detaching execution. Threads delegate work to the thread currently holding the lock and are able to detach , i.e., immediately continue their execution until they need a result from a previously delegated critical section. We show how to use queue delegation to build synchronization algorithms with lower overhead and higher throughput than existing algorithms, even when critical sections need to communicate results back immediately. Experiments when using up to 64 threads to access a shared priority queue show that QD locking provides 10 times higher throughput than Pthreads mutex locks and outperforms leading delegation algorithms. Also, when mixing parallel reads with delegated write operations, QD locking outperforms competing algorithms with an advantage ranging from 9.5 up to 207 percent increased throughput. Last but not least, continuing execution instead of waiting for the execution of critical sections leads to increased parallelism and better scalability. As we will see, queue delegation locking uses simple building blocks whose overhead is low even in uncontended use. All these make the technique useful in a wide variety of applications.

Benchmark coupled-cluster g-tensor calculations with full inclusion of the two-particle spin-orbit contributions.

We present a parallel implementation to compute electron spin resonance g-tensors at the coupled-cluster singles and doubles (CCSD) level which employs the ACES III domain-specific software tools for scalable parallel programming, i.e., the super instruction architecture language and processor (SIAL and SIP), respectively. A unique feature of the present implementation is the exact (not approximated) inclusion of the five one- and two-particle contributions to the g-tensor [i.e., the mass correction, one- and two-particle paramagnetic spin-orbit, and one- and two-particle diamagnetic spin-orbit terms]. Like in a previous implementation with effective one-electron operators [J. Gauss et al., J. Phys. Chem. A 113, 11541-11549 (2009)], our implementation utilizes analytic CC second derivatives and, therefore, classifies as a true CC linear-response treatment. Therefore, our implementation can unambiguously appraise the accuracy of less costly effective one-particle schemes and provide a rationale for their widespread use. We have considered a large selection of radicals used previously for benchmarking purposes including those studied in earlier work and conclude that at the CCSD level, the effective one-particle scheme satisfactorily captures the two-particle effects less costly than the rigorous two-particle scheme. With respect to the performance of density functional theory (DFT), we note that results obtained with the B3LYP functional exhibit the best agreement with our CCSD results. However, in general, the CCSD results agree better with the experimental data than the best DFT/B3LYP results, although in most cases within the rather large experimental error bars.

Ultrafast scalable parallel algorithm for the radial distribution function histogramming using MPI maps

In the present paper, an ultrafast and scalable parallel program for the Radial Pair Function (RPF) is presented via pure Message Passing Interface (MPI) paradigm. The parallel code computes the radial distribution function for the single-component as well as multi-component systems. The single-component and multi-component systems have been extracted for benchmarking purposes by means of user-written codes in C++ for the Graphite structure and MPI C++ for hydrogen adsorption via Grand Canonical Monte Carlo (GCMC) in the Single-Walled Carbon Nanotube (SWNT), respectively. The speedup and efficiency curves substantiate an excellent performance in terms of computing time and computation size as well. Additionally, the mentioned MPI paradigms are nearly five times (single-component systems) and two times (multi-component systems) faster than the relevant parallel codes using a machine with 48 CPUs and NVIDIA Quadro K5200/PCIe/SSE2. Some conclusions and outlooks pertaining to the numerical implementations, algorithm optimization involving the space decomposition idea have been discussed and provided.

A task-uncoordinated distributed dataflow model for scalable high performance parallel program execution

We describe a novel model for executing distributed memory parallel programs using uncoordinated tasks.We describe several off-line optimizations for the proposed model.We examine the effects of these optimizations on modern processors with wider vector units.Increasing levels of task coalescence can improve throughput and increase performance.Increases in performance are observed in both single node and multi node experiments. We propose a distributed dataflow execution model which utilizes a distributed dictionary for data memoization, allowing each parallel task to schedule instructions without direct inter-task coordination. We provide a description of the proposed model, including autonomous dataflow task selection. We also describe a set of optimization strategies which improve overall throughput of stencil programs executed using this model on modern multi-core and vectorized architectures.

Implementation of GPU virtualization using PCI pass-through mechanism

As a general purpose scalable parallel programming model for coding highly parallel applications, CUDA from NVIDIA provides several key abstractions: a hierarchy of thread blocks, shared memory, and barrier synchronization. It has proven to be rather effective at programming multithreaded many-core GPUs that scale transparently to hundreds of cores; as a result, scientists all over the industry and academia are using CUDA to dramatically expedite on production and codes. GPU-based clusters are likely to play an essential role in future cloud computing centers, because some computation-intensive applications may require GPUs as well as CPUs. In this paper, we adopted the PCI pass-through technology and set up virtual machines in a virtual environment; thus, we were able to use the NVIDIA graphics card and the CUDA high performance computing as well. In this way, the virtual machine has not only the virtual CPU but also the real GPU for computing. The performance of the virtual machine is predicted to increase dramatically. This paper measured the difference of performance between physical and virtual machines using CUDA, and investigated how virtual machines would verify CPU numbers under the influence of CUDA performance. At length, we compared CUDA performance of two open source virtualization hypervisor environments, with or without using PCI pass-through. Through experimental results, we will be able to tell which environment is most efficient in a virtual environment with CUDA.

MPI + MPI: a new hybrid approach to parallel programming with MPI plus shared memory

Hybrid parallel programming with the message passing interface (MPI) for internode communication in conjunction with a shared-memory programming model to manage intranode parallelism has become a dominant approach to scalable parallel programming. While this model provides a great deal of flexibility and performance potential, it saddles programmers with the complexity of utilizing two parallel programming systems in the same application. We introduce an MPI-integrated shared-memory programming model that is incorporated into MPI through a small extension to the one-sided communication interface. We discuss the integration of this interface with the MPI 3.0 one-sided semantics and describe solutions for providing portable and efficient data sharing, atomic operations, and memory consistency. We describe an implementation of the new interface in the MPICH2 and Open MPI implementations and demonstrate an average performance improvement of 40 % to the communication component of a five-point stencil solver.

Parallel programming patterns for multi-processor SoC

Efficient, scalable and productive parallel programming is a major challenge for exploiting the future multi-processor SoC platforms. This article presents the MultiFlex programming environment which has been developed to address this challenge. It is targeted for use on Platform 2012 , a scalable multi-processor fabric. The MultiFlex environment supports high-level simulation, iterative platform mapping, and includes tools for programming model aware debug, trace, visualization and analysis. This article focuses on the two classes of programming abstractions supported in MultiFlex. The first is a set of Parallel Programming Patterns (PPP) which offer a rich set of programming abstractions for implementing efficient data- and task-level parallel applications. The second is a Reactive Task Management (RTM) abstraction, which offers a lightweight C-based API to support dynamic dispatching of small grain tasks on tightly coupled parallel processing resources. The use of the MultiFlex native programming model is illustrated through the capture and mapping of two representative video applications. The first is a high-quality rescaling (HQR) application on a multi-processor platform. We present the details of the optimization process which was required for mapping the HQR application, for which the reference code requires 350 GIPS (giga instructions per second), onto a 16 processor cluster. Our results show that the parallel implementation using the PPP model offers almost linear acceleration with respect to the number of processing elements. The second application is a high-definition VC-1 decoder. For this application, we illustrate two different parallel programming model variants, one using PPPs, the other based on RTM. These two versions are mapped onto two variants of a homogeneous version of the Platform 2012 multi-core fabric.

READSCAN: a fast and scalable pathogen discovery program with accurate genome relative abundance estimation

Summary: READSCAN is a highly scalable parallel program to identify non-host sequences (of potential pathogen origin) and estimate their genome relative abundance in high-throughput sequence datasets. READSCAN accurately classified human and viral sequences on a 20.1 million reads simulated dataset in <27 min using a small Beowulf compute cluster with 16 nodes (Supplementary Material). Availability: http://cbrc.kaust.edu.sa/readscan Contact: arnab.pain@kaust.edu.sa or raeece.naeem@gmail.comSupplementary information: Supplementary data are available at Bioinformatics online.

Editorial: enabling technologies for programming extreme scale systems

Multi-core architecture presents a new trend, and core-based parallel processing algorithms will continue to become more important. In addition, Grid and Cloud Computing (GCC) has emerged rapidly as an exciting new paradigm that offers a challenging model of computing and poses fascinating problems ranging from parallel architecture to programming models and compiler optimization. Furthermore, the wide-scale deployment of computing nodes, clusters, and embedded processors will continue to fuel core parallel processing and distributed systems algorithms. Thus, the importance of scalable parallel programming will continue to grow. The field of parallel and distributed computing is going through rapid changes. The emerging multi-core commodity processors and advanced networking technologies are allowing for the design of cost-effective parallel and distributed systems. To achieve scalability and high performance in these systems, it is increasingly becoming challenging to design better architecture, software environments, networking protocols, compilers, operating systems, languages, algorithms and applications for designing next generation scalable and high-performance parallel systems. This special issue is intended to foster stateof-the-art research in the area of extreme scale parallel systems including the topics of parallel algorithm design, data flow analysis, virtualization, energy efficient, load balancing, parallel computing model, multi-core programming, GPU implementation and optimization, execution on real-world parallel architecture and novel applications associated with this new paradigm. The papers included in this special issue demonstrate the effectiveness and efficiency of a variety of technologies and applications in parallel, grid, cloud and extreme scale systems. All of them not only contribute valuable insights and results but

Hybrid CUDA, OpenMP, and MPI parallel programming on multicore GPU clusters

Nowadays, NVIDIA's CUDA is a general purpose scalable parallel programming model for writing highly parallel applications. It provides several key abstractions – a hierarchy of thread blocks, shared memory, and barrier synchronization. This model has proven quite successful at programming multithreaded many core GPUs and scales transparently to hundreds of cores: scientists throughout industry and academia are already using CUDA to achieve dramatic speedups on production and research codes. In this paper, we propose a parallel programming approach using hybrid CUDA OpenMP, and MPI programming, which partition loop iterations according to the number of C1060 GPU nodes in a GPU cluster which consists of one C1060 and one S1070. Loop iterations assigned to one MPI process are processed in parallel by CUDA run by the processor cores in the same computational node.

Development of parallel software for resolving the 3D task about the origin of the Tornado according to the Nikolaevskii theory

The aim of the paper is to develop a scalable parallel program for numerical solution of the equation set that simulates processes and origin conditions of intensive atmospheric vortices (IAV) in 3D compressible atmosphere based on the theory of mesovortice turbulence by Nikolaevskii. The original system of nonlinear differential equations and its boundary and initial conditions are considered in the paper. The difference scheme with required precision is selected. The parameters’ restrictions are also formulated. The structure of parallel software that performs computations on a high-performance cluster is considered. The problems of the software’s optimization in order to improve its scalability are discussed. The results of numerical calculations are considered in summary.

The third homomorphism theorem on trees

Parallel programs on lists have been intensively studied. It is well known that associativity provides a good characterization for divide-and-conquer parallel programs. In particular, the third homomorphism theorem is not only useful for systematic development of parallel programs on lists, but it is also suitable for automatic parallelization. The theorem states that if two sequential programs iterate the same list leftward and rightward, respectively, and compute the same value, then there exists a divide-and-conquer parallel program that computes the same value as the sequential programs. While there have been many studies on lists, few have been done for characterizing and developing of parallel programs on trees. Naive divide-and-conquer programs, which divide a tree at the root and compute independent subtrees in parallel, take time that is proportional to the height of the input tree and have poor scalability with respect to the number of processors when the input tree is ill-balanced. In this paper, we develop a method for systematically constructing scalable divide-and-conquer parallel programs on trees, in which two sequential programs lead to a scalable divide-andconquer parallel program. We focus on paths instead of trees so as to utilize rich results on lists and demonstrate that associativity provides good characterization for scalable divide-and-conquer parallel programs on trees. Moreover, we generalize the third homomorphism theorem from lists to trees.We demonstrate the effectiveness of our method with various examples. Our results, being generalizations of known results for lists, are generic in the sense that they work well for all polynomial data structures.

Tools for scalable parallel program analysis: Vampir NG, MARMOT, and DeWiz

Large-scale high-performance computing systems pose a tough obstacle for today's program analysis tools. Their demands in computational performance and memory capacity for processing program analysis data exceed the capabilities of standard workstations and traditional analysis tools. A comparison of the sophisticated approaches of Vampir NG (VNG), the Debugging Wizard DeWiz and the correctness-checking tool MARMOT provides novel ideas for scalable parallel program analysis. While VNG exploits the power of cluster architectures for near real-time performance analysis, DeWiz utilises distributed computing infrastructures for distinct analysis activities. MARMOT combines automatic runtime and partially distributed analysis.

Scalable Parallel Programming with CUDA

The advent of multicore CPUs and manycore GPUs means that mainstream processor chips are now parallel systems. Furthermore, their parallelism continues to scale with Moore’s law. The challenge is to develop mainstream application software that transparently scales its parallelism to leverage the increasing number of processor cores, much as 3D graphics applications transparently scale their parallelism to manycore GPUs with widely varying numbers of cores.