Complexity Of Parallel Programming Research Articles

A tale of lock-free agents: towards Software Transactional Memory in parallel Agent-Based Simulation

With the decline of Moore’s law and the ever increasing availability of cheap massively parallel hardware, it becomes more and more important to embrace parallel programming methods to implement Agent-Based Simulations (ABS). This has been acknowledged in the field a while ago and numerous research on distributed parallel ABS exists, focusing primarily on Parallel Discrete Event Simulation as the underlying mechanism. However, these concepts and tools are inherently difficult to master and apply and often an excess in case implementers simply want to parallelise their own, custom agent-based model implementation. However, with the established programming languages in the field, Python, Java and C++, it is not easy to address the complexities of parallel programming due to unrestricted side effects and the intricacies of low-level locking semantics. Therefore, in this paper we propose the use of a lock-free approach to parallel ABS using Software Transactional Memory (STM) in conjunction with the pure functional programming language Haskell, which in combination, removes some of the problems and complexities of parallel implementations in imperative approaches. We present two case studies, in which we compare the performance of lock-based and lock-free STM implementations in two different well known Agent-Based Models, where we investigate both the scaling performance under increasing number of CPU cores and the scaling performance under increasing number of agents. We show that the lock-free STM implementations consistently outperform the lock-based ones and scale much better to increasing number of CPU cores both on local hardware and on Amazon EC. Further, by utilizing the pure functional language Haskell we gain the benefits of immutable data and lack of unrestricted side effects guaranteed at compile-time, making validation easier and leading to increased confidence in the correctness of an implementation, something of fundamental importance and benefit in parallel programming in general and scientific computing like ABS in particular.

Visualizing Hierarchical Performance Profiles of Parallel Codes Using CallFlow.

Calling context trees (CCTs) couple performance metrics with call paths, helping understand the execution and performance of parallel programs. To identify performance bottlenecks, programmers and performance analysts visually explore CCTs to form and validate hypotheses regarding degraded performance. However, due to the complexity of parallel programs, existing visual representations do not scale to applications running on a large number of processors. We present CallFlow, an interactive visual analysis tool that provides a high-level overview of CCTs together with semantic refinement operations to progressively explore CCTs. Using a flow-based metaphor, we visualize a CCT by treating execution time as a resource spent during the call chain, and demonstrate the effectiveness of our design with case studies on large-scale, production simulation codes.

Parallel bitsliced AES through PHAST: a single-source high-performance library for multi-cores and GPUs

PHAST library is a high-level heterogeneous STL-like C $$++$$ library that can be targeted on multi-core processors and Nvidia GPUs. It permits to exploit the performance of modern parallel architectures without the complexity of parallel programming. The library manages the programming and critical fine tuning of the parallel execution on both platforms without interfering with the application code structure, while maintaining the possibility to use architecture-specific features and instructions. In cryptography, performance and architectural efficiency of software implementations is crucial. This is witnessed by the extensive research in highly optimized and specialized versions of many protocols. In this paper, we assess the performance overhead and productivity improvement achievable through the PHAST library. We implement a pseudo random number generator (PRNG) based on cache-timing-attack resistant AES. We compare it with the fastest implementations in both CPU and Nvidia GPU domains. Achieved results show that the PHAST code is shorter and simpler than the state-of-the-art implementations. Its source length is 59.59% of the reference CUDA C implementation and 88.18% of the sequential C $$++$$ version for CPUs, despite being the same for both targets. It is also far less complex in terms of McCabe’s and Halstead’s metrics. Results show that these productivity improvements induce a limited performance overhead of the library layer: less than 5% on single-thread execution for CPUs and around 10% on Nvidia GPUs. Furthermore, performance of the PHAST PRNG automatically scales with the available cores in a nearly linear fashion, allowing programmers to fully exploit multi-core resources with the same source code.

Space-based parallel program design process with high-level communication channels

In this research, we reduce design complexity of parallel programs by performing the design task at the high level. We devised a formalism of the parallel program design process to support space-based parallel program design. In this formalism, program designs are captured in entity spaces. Entity spaces, including data spaces, tasks, etc. can be subdivided and freely distributed into logical node clusters that are assigned to processes. We designed high-level communication channels for communication among divided subtasks in a parallel program for space-based communication integration. Supported program models range in single program multiple data, multiple program multiple data, and their hybrid model. We designed single-space and cross-space high-level communication channels with various communication topologies. They are specified externally and decoupled from a parallel program. This makes the program logical and easily designed and reused. A practical example is used for illustrating the design process.

Al3c: high-performance software for parameter inference using Approximate Bayesian Computation.

The development of Approximate Bayesian Computation (ABC) algorithms for parameter inference which are both computationally efficient and scalable in parallel computing environments is an important area of research. Monte Carlo rejection sampling, a fundamental component of ABC algorithms, is trivial to distribute over multiple processors but is inherently inefficient. While development of algorithms such as ABC Sequential Monte Carlo (ABC-SMC) help address the inherent inefficiencies of rejection sampling, such approaches are not as easily scaled on multiple processors. As a result, current Bayesian inference software offerings that use ABC-SMC lack the ability to scale in parallel computing environments. We present al3c, a C++ framework for implementing ABC-SMC in parallel. By requiring only that users define essential functions such as the simulation model and prior distribution function, al3c abstracts the user from both the complexities of parallel programming and the details of the ABC-SMC algorithm. By using the al3c framework, the user is able to scale the ABC-SMC algorithm in parallel computing environments for his or her specific application, with minimal programming overhead. al3c is offered as a static binary for Linux and OS-X computing environments. The user completes an XML configuration file and C++ plug-in template for the specific application, which are used by al3c to obtain the desired results. Users can download the static binaries, source code, reference documentation and examples (including those in this article) by visiting https://github.com/ahstram/al3c. astram@usc.edu Supplementary data are available at Bioinformatics online.

An Auto-Matching Model with Pattern Recognition Using Bayesian Classifier for Parallel Programming on A Multi-Core Processor

The emerging multi-core processor architecture has greatly escalated scientific computing, but, at the same time, made parallel programming increasingly complex and challenging. In this paper, the use of the Auto Parallel Classification (APC) model in an Object-Oriented Parallel Model (OOPModel) environment is demonstrated. A designed module provides a traversal and a reduction of the DAG task graph. The parallel characteristics vectors, which are analyzed according to Naive Bayesian classification theory, are critical parameters for matching and generating parallel design patterns and various skeletal frameworks. Through extensive experimentation, it is demonstrated, that by using the Map-Reduce pattern to develop a minimum-sort algorithm, in conjunction with the APC model, we can achieve a reduction in the complexity of parallel programming and the minimization of errors. Most importantly, through scientific experimentation, this document will further demonstrate that correct computational results and movements toward linear speed-up can be accomplished.

An Optimized High-Throughput Strategy for Constructing Inverted Files

Current high-throughput algorithms for constructing inverted files all follow the MapReduce framework, which presents a high-level programming model that hides the complexities of parallel programming. In this paper, we take an alternative approach and develop a novel strategy that exploits the current and emerging architectures of multicore processors. Our algorithm is based on a high-throughput pipelined strategy that produces parallel parsed streams, which are immediately consumed at the same rate by parallel indexers. We have performed extensive tests of our algorithm on a cluster of 32 nodes, and were able to achieve a throughput close to the peak throughput of the I/O system: a throughput of 280 MB/s on a single node and a throughput that ranges between 5.15 GB/s (1 Gb/s Ethernet interconnect) and 6.12 GB/s (10 Gb/s InfiniBand interconnect) on a cluster with 32 nodes for processing the ClueWeb09 data set. Such a performance represents a substantial gain over the best known MapReduce algorithms even when comparing the single node performance of our algorithm to MapReduce algorithms running on large clusters. Our results shed a light on the extent of the performance cost that may be incurred by using the simpler, higher level MapReduce programming model for large scale applications.

PRPL

pRPL is an open-source general-purpose programming library developed by the author to parallelize almost any raster-processing algorithm with any arbitrary neighborhood configuration, and support any data type. This paper introduces the advanced features of pRPL, compares it with other similar programming libraries, and demonstrates the performance of a parallel geographic Cellular Automata (CA) model developed using pRPL with real-world datasets. In conclusion, pRPL effectively reduces the development complexity of parallel programming, and efficiently reduces the computing time.

An efficient transactional memory algorithm for computing minimum spanning forest of sparse graphs

Due to power wall, memory wall, and ILP wall, we are facing the end of ever increasing single-threaded performance. For this reason, multicore and manycore processors are arising as a new paradigm to pursue. However, to fully exploit all the cores in a chip, parallel programming is often required, and the complexity of parallel programming raises a significant concern. Data synchronization is a major source of this programming complexity, and Transactional Memory is proposed to reduce the difficulty caused by data synchronization requirements, while providing high scalability and low performance overhead. The previous literature on Transactional Memory mostly focuses on architectural designs. Its impact on algorithms and applications has not yet been studied thoroughly. In this paper, we investigate Transactional Memory from the algorithm designer's perspective. This paper presents an algorithmic model to assist in the design of efficient Transactional Memory algorithms and a novel Transactional Memory algorithm for computing a minimum spanning forest of sparse graphs. We emphasize multiple Transactional Memory related design issues in presenting our algorithm. We also provide experimental results on an existing software Transactional Memory system. Our algorithm demonstrates excellent scalability in the experiments, but at the same time, the experimental results reveal the clear limitation of software Transactional Memory due to its high performance overhead. Based on our experience, we highlight the necessity of efficient hardware support for Transactional Memory to realize the potential of the technology.

On Data Distributions in the Construction of Parallel Programs

Data distributions have a serious impact on time complexity of parallel programs, developed based on domain decomposition. A new kind of distributions—set distributions, based on set-valued mappings, is introduced. These distributions assign a data object to more than one process. The set distributions can be used especially when the number of processes is greater than the data input size, but, sometimes using set distributions can lead to efficient general parallel algorithms. The work-load properties of these distributions and their impact on the number of communications are discussed. In order to illustrate the implications of data distributions in the construction of parallel programs, some examples are presented. Two parallel algorithms for computation of Lagrange interpolation polynomial are developed, starting from simple distributions and set distributions.

CALCULATING AN OPTIMAL HOMOMORPHIC ALGORITHM FOR BRACKET MATCHING

It is widely recognized that a key problem of parallel computation is in the development of both efficient and correct parallel software. Although many advanced language features and compilation techniques have been proposed to alleviate the complexity of parallel programming, much effort is still required to develop parallelism in a formal and systematic way. In this paper, we intend to clarify this point by demonstrating a formal derivation of a correct but efficient homomorphic parallel algorithm for a simple language recognition problem known as bracket matching. To the best of our knowledge, our formal derivation leads to a novel divide-and-conquer parallel algorithm for bracket matching.

Design of an Actor language for implicit parallel programming

Software tools that support implicit parallel programming hold the key to reducing the complexity of parallel programming and realizing more ubiquitous parallel computation. This paper addresses the topic of implicit concurrent object-oriented programming. It presents a new language, SYMPAL, which unlike most existing COOP languages that provide explicit constructs for concurrency control, is based on a unification of object orientation and pure functional programming with the goals of supporting implicit programming and high efficiency in massively concurrent object-oriented programming. Extensive experience with SYMPAL applications on a parallel machine indicates that it achieves these goals.

Experience with parallel programming using code templates

For almost a decade we have been working at developing and using template-based models for parallel computing. Template-based models separate the specification of the parallel structuring aspects from the application code that is to be parallelized. A user provides the application code and specifies the parallel structure of the application using high-level icons, called templates. The parallel programming system then generates the code necessary for parallelizing the application. The goal here is to provide a mechanism for quick and reliable development of coarse-grain parallel applications that employ frequently occurring parallel structures. Our initial template-based system, FrameWorks, was positively received but had a number of shortcomings. The Enterprise parallel programming environment evolved out of this work. Now, after several years of experience with the system, its shortcomings are becoming evident. Controlled experiments have been conducted to assess the usability of our system in comparison with other systems. The paper outlines our experiences in developing and using these systems. A list of desirable characteristics of template-based models is given. The FrameWorks and Enterprise systems are discussed in the context of these characteristics and the results of our usability experiments. Many of our observations are relevant to other parallel programming systems, even though they may be based on different assumptions. Although template-base models have the potential for simplifying the complexities of parallel programming, they have yet to realize these expectations for high-performance applications. © 1998 John Wiley & Sons, Ltd.

Experience with parallel programming using code templates

For almost a decade we have been working at developing and using template-based models for parallel computing. Template-based models separate the specification of the parallel structuring aspects from the application code that is to be parallelized. A user provides the application code and specifies the parallel structure of the application using high-level icons, called templates. The parallel programming system then generates the code necessary for parallelizing the application. The goal here is to provide a mechanism for quick and reliable development of coarse-grain parallel applications that employ frequently occurring parallel structures. Our initial template-based system, FrameWorks, was positively received but had a number of shortcomings. The Enterprise parallel programming environment evolved out of this work. Now, after several years of experience with the system, its shortcomings are becoming evident. Controlled experiments have been conducted to assess the usability of our system in comparison with other systems. The paper outlines our experiences in developing and using these systems. A list of desirable characteristics of template-based models is given. The FrameWorks and Enterprise systems are discussed in the context of these characteristics and the results of our usability experiments. Many of our observations are relevant to other parallel programming systems, even though they may be based on different assumptions. Although template-base models have the potential for simplifying the complexities of parallel programming, they have yet to realize these expectations for high-performance applications. © 1998 John Wiley & Sons, Ltd.

Multiprocessors from a software perspective

Like many architectural techniques that originated with mainframes. the use of multiple processors in a single computer is becoming popular in workstations and even personal computers. Multiprocessors constitute a significant percentage of recent workstation sales, and highly affordable multiprocessor personal computers are available in local computer stores. Once again, we find ourselves in a familiar situation: hardware is ahead of software. Because of the complexity of parallel programming, multiprocessors today are rarely used to speed up individual applications. Instead, they usually function as cycle-servers that achieve increased system throughput by running multiple tasks simultaneously. Automatic parallelization by a compiler is a particularly attractive approach to software development for multiprocessors, as it enables ordinary sequential programs to take advantage of the multiprocessor hardware without user involvement. This article looks to the future by examining some of the latest research results in automatic parallelization technology.

PERFORMANCE EVALUATION OF BACKPROPAGATION NEURAL NETWORK IMPLEMENTATION ON SINGLE PROGRAM MULTIPLE DATA (SPMD) ARCHITECTURE

The Single Program Multiple Data (SPMD) approach has been developed to reduce the system overheads, to entail minimal computation time and to reduce the programming complexity of parallel programs. The SPMD computational model is based on the premise that all the processes execute the same program; however, at any instant of time, these processes may be executing different instructions and may be operating on different data. This work uses the SPMD computational model to implement a backpropagation neural network and the implementation is done on a multiprocessor. We present the distributed backpropagation forward and backward execution algorithms in SPMD mode. The performance of the SPMD-implemented ANN is demonstrated by the Automatic Target Recognition (ATR) problem, which involves identification of a class of targets depending on a set of input data values. The performance of the SPMD-implemented ANN is studied by observing the speedups achieved for training times as the number of processors is increa...

The semantics and complexity of parallel programs for vector computations. Part I: A case study using ADA

Recent research in parallel numerical computation has tended to focus on the algorithmic level. Less attention has been given to the programming level where algorithm is matched, to some extent, to computer architecture. This two-part paper presents a three-level approach to parallel programming which distinguishes between mathematical algorithm, program and computer architecture. In part I, we motivate our approach by a case study using the Ada language. In part II, a mathematical concept of parallel algorithm is introduced in terms of partial orders. This serves as the basis of a theory of parallel computation which makes possible a precise semantics and a precise criterion of complexity of parallel programs. It also suggests some notation for specifying parallel numerical algorithms. To illustrate the ideas presented in part II, we concentrate here on parallel numerical computations which have vector spaces as their central data type and which are intended to be executed on a multi-processor system. The Ada language, with its task constructs, allows one to program computer algorithms to be executed on multi-processor systems, rather than on “vector (pipelined) architectures”. To provide a concrete example of the general problem of programming parallel numerical algorithms for multi-processor computers, we do a case study of how Ada can be used to program the solution of a system of linear equations on such computers. The case study includes an analysis of complexity which addresses the cost of data movement and process control/synchronization as well as the usual arithmetic complexity.