Parallel Programming Methodology Research Articles

Accelerating Fortran codes: A method for integrating Coarray Fortran with CUDA Fortran and OpenMP

Fortran's prominence in scientific computing requires strategies to ensure both that legacy codes are efficient on high-performance computing systems, and that the language remains attractive for the development of new high-performance codes. Coarray Fortran (CAF), part of the Fortran 2008 standard introduced for parallel programming, facilitates distributed memory parallelism with a syntax familiar to Fortran programmers, simplifying the transition from single-processor to multi-processor coding. This research focuses on innovating and refining a parallel programming methodology that fuses the strengths of Intel Coarray Fortran, Nvidia CUDA Fortran, and OpenMP for distributed memory parallelism, high-speed GPU acceleration and shared memory parallelism respectively. We consider the management of pageable and pinned memory, CPU-GPU affinity in NUMA multiprocessors, and robust compiler interfacing with speed optimisation. We demonstrate our method through its application to a parallelised Poisson solver and compare the methodology, implementation, and scaling performance to that of the Message Passing Interface (MPI), finding CAF offers similar speeds with easier implementation. For new codes, this approach offers a faster route to optimised parallel computing. For legacy codes, it eases the transition to parallel computing, allowing their transformation into scalable, high-performance computing applications without the need for extensive re-design or additional syntax.

The NAS Parallel Benchmarks for evaluating C++ parallel programming frameworks on shared-memory architectures

The NAS Parallel Benchmarks (NPB), originally implemented mostly in Fortran, is a consolidated suite containing several benchmarks extracted from Computational Fluid Dynamics (CFD) models. The benchmark suite has important characteristics such as intensive memory communications, complex data dependencies, different memory access patterns, and hardware components/sub-systems overload. Parallel programming APIs, libraries, and frameworks that are written in C++ as well as new optimizations and parallel processing techniques can benefit if NPB is made fully available in this programming language. In this paper we present NPB-CPP, a fully C++ translated version of NPB consisting of all the NPB kernels and pseudo-applications developed using OpenMP, Intel TBB, and FastFlow parallel frameworks for multicores. The design of NPB-CPP leverages the Structured Parallel Programming methodology (essentially based on parallel design patterns). We show the structure of each benchmark application in terms of composition of few patterns (notably Map and MapReduce constructs) provided by the selected C++ frameworks. The experimental evaluation shows the accuracy of NPB-CPP with respect to the original NPB source code. Furthermore, we carefully evaluate the parallel performance on three multi-core systems (Intel, IBM Power, and AMD) with different C++ compilers (gcc, icc, and clang) by discussing the performance differences in order to give to the researchers useful insights to choose the best parallel programming framework for a given type of problem.

Algorithmic Skeletons and Parallel Design Patterns in Mainstream Parallel Programming

This paper discusses the impact of structured parallel programming methodologies in state-of-the-art industrial and research parallel programming frameworks. We first recap the main ideas underpinning structured parallel programming models and then present the concepts of algorithmic skeletons and parallel design patterns. We then discuss how such concepts have permeated the wider parallel programming community. Finally, we give our personal overview—as researchers active for more than two decades in the parallel programming models and frameworks area—of the process that led to the adoption of these concepts in state-of-the-art industrial and research parallel programming frameworks, and the perspectives they open in relation to the exploitation of forthcoming massively-parallel (both general and special-purpose) architectures.

Bringing Parallel Patterns Out of the Corner

High-level parallel programming is an active research topic aimed at promoting parallel programming methodologies that provide the programmer with high-level abstractions to develop complex parallel software with reduced time to solution. Pattern-based parallel programming is based on a set of composable and customizable parallel patterns used as basic building blocks in parallel applications. In recent years, a considerable effort has been made in empowering this programming model with features able to overcome shortcomings of early approaches concerning flexibility and performance. In this article, we demonstrate that the approach is flexible and efficient enough by applying it on 12 out of 13 PARSEC applications. Our analysis, conducted on three different multicore architectures, demonstrates that pattern-based parallel programming has reached a good level of maturity, providing comparable results in terms of performance with respect to both other parallel programming methodologies based on pragma-based annotations (i.e., Open mp and O mp S s ) and native implementations (i.e., P threads ). Regarding the programming effort, we also demonstrate a considerable reduction in lines of code and code churn compared to P threads and comparable results with respect to other existing implementations.

Lazy Scheduling

Lazy scheduling is a runtime scheduler for task-parallel codes that effectively coarsens parallelism on load conditions in order to significantly reduce its overheads compared to existing approaches, thus enabling the efficient execution of more fine-grained tasks. Unlike other adaptive dynamic schedulers, lazy scheduling does not maintain any additional state to infer system load and does not make irrevocable serialization decisions. These two features allow it to scale well and to provide excellent load balancing in practice but at a much lower overhead cost compared to work stealing, the golden standard of dynamic schedulers. We evaluate three variants of lazy scheduling on a set of benchmarks on three different platforms and find it to substantially outperform popular work stealing implementations on fine-grained codes. Furthermore, we show that the vast performance gap between manually coarsened and fully parallel code is greatly reduced by lazy scheduling, and that, with minimal static coarsening, lazy scheduling delivers performance very close to that of fully tuned code. The tedious manual coarsening required by the best existing work stealing schedulers and its damaging effect on performance portability have kept novice and general-purpose programmers from parallelizing their codes. Lazy scheduling offers the foundation for a declarative parallel programming methodology that should attract those programmers by minimizing the need for manual coarsening and by greatly enhancing the performance portability of parallel code.

DNA Codeword Design: Theory and Applications

This is a survey of the origin, current progress and applications of a major roadblock to the development of analytic models for DNA computing (a massively parallel programming methodology) and DNA self-assembly (a nanofabrication methodology), namely the so-called CODEWORD DESIGN problem. The problem calls for finding large sets of single DNA strands that do not crosshybridize to themselves or to their complements and has been recognized as an important problem in DNA computing, self-assembly, DNA memories and phylogenetic analyses because of their error correction and prevention properties. Major recent advances include the development of experimental techniques to search for such codes, as well as a theoretical framework to analyze this problem, despite the fact that it has been proven to be NP-complete using any single concrete metric space to model the Gibbs energy. In this framework, codeword design is reduced to finding large sets of strands maximally separated in DNA spaces and, therefore, the key to finding such sets would lie in knowledge of the geometry of these spaces. A new general technique has been recently found to embed them in Euclidean spaces in a hybridization-affinity-preserving manner, i.e., in such a way that oligos with high/low hybridization affinity are mapped to neighboring/remote points in a geometric lattice, respectively. This isometric embedding materializes long-held metaphors about codeword design in terms of sphere packing and error-correcting codes and leads to designs that are in some cases known to be provably nearly optimal for some oligo sizes. It also leads to upper and lower bounds on estimates of the size of optimal codes of size up to 32–mers, as well as to infinite families of solutions to CODEWORD DESIGN, based on estimates of the kissing (or contact) number for sphere packings in Euclidean spaces. Conversely, this reduction suggests interesting new algorithms to find dense sphere packing solutions in high dimensional spheres using results for CODEWORD DESIGN previously obtained by experimental or theoretical molecular means, as well as a proof that finding these bounds exactly is NP-complete in general. Finally, some research problems and applications arising from these results are described that might be of interest for further research.

Towards a systematic approach to the dynamic adaptation of structured parallel computations using model predictive control

Adaptiveness is an essential feature for distributed parallel applications executed on dynamic environments like Grids and Clouds. Being adaptive means that parallel components can change their configuration at run-time (by modifying their parallelism degree or switching to a different parallel variant) to face irregular workload or to react to uncontrollable changes of the execution platform. A critical problem consists in the definition of adaptation strategies able to select optimal reconfigurations (minimizing operating costs and reconfiguration overhead) and achieve the stability of control decisions (avoiding unnecessary reconfigurations). This paper presents an approach to apply Model Predictive Control (a form of optimal control studied in Control Theory) to adaptive parallel computations expressed according to the Structured Parallel Programming methodology. We show that predictive control is amenable to achieve stability and optimality by relying on the predictability of structured parallelism patterns and the possibility to express analytical cost models of their QoS metrics. The approach has been exemplified on two case-studies, providing a first assessment of its potential and feasibility.

An execution environment for flexible task-oriented software on multicore systems

The article addresses the challenges of software development for current and future parallel computers, which are expected to be dominated by multicore and many-core architectures. Using these multicore processors for cluster systems will create systems with thousands of cores and deep memory hierarchies. To efficiently exploit the tremendous parallelism of these hardware platforms, a new generation of programming methodologies is needed. This article proposes a parallel programming methodology exploiting a task-based representation of application software. For the specification of task-based programs, a coordination language is presented, which uses external variables to express the cooperation between tasks. For the actual execution of a task-based program on a specific parallel architecture, different dynamic scheduling algorithms embedded into an execution environment are introduced. Runtime experiments for complex methods from a numerical analysis are performed on different parallel execution platforms.

A Parallel Mixed Integer Programming-Finite Element Method Technique for Global Design Optimization of Power Transformers

Transformer design optimization is determined by minimizing the transformer cost taking into consideration constraints imposed both by international specifications and customer needs. The main purpose of this work is the development and validation of an optimization technique based on a parallel mixed integer nonlinear programming methodology in conjunction with the finite element method, in order to reach a global optimum design of wound core power transformers. The proposed optimization methodology has been implemented into software able to provide a global feasible solution for every given set of initial values for the design variables, rendering it suitable for application in the industrial transformer design environment.

High-performance data mining with skeleton-based structured parallel programming

We show how to apply a structured parallel programming (SPP) methodology based on skeletons to data mining (DM) problems, reporting several results about three commonly used mining techniques, namely association rules, decision tree induction and spatial clustering. We analyze the structural patterns common to these applications, looking at application performance and software engineering efficiency. Our aim is to clearly state what features a SPP environment should have to be useful for parallel DM. Within the skeleton-based PPE SkIE that we have developed, we study the different patterns of data access of parallel implementations of Apriori, C4.5 and DBSCAN. We need to address large partitions reads, frequent and sparse access to small blocks, as well as an irregular mix of small and large transfers, to allow efficient development of applications on huge databases. We examine the addition of an object/component interface to the skeleton structured model, to simplify the development of environment-integrated, parallel DM applications.

Parallel Programming Environment for OpenMP

We present our effort to provide a comprehensive parallel programming environment for the OpenMP parallel directive language. This environment includes a parallel programming methodology for the OpenMP programming model and a set of tools (Ursa Minor and InterPol) that support this methodology. Our toolset provides automated and interactive assistance to parallel programmers in time‐consuming tasks of the proposed methodology. The features provided by our tools include performance and program structure visualization, interactive optimization, support for performance modeling, and performance advising for finding and correcting performance problems. The presented evaluation demonstrates that our environment offers significant support in general parallel tuning efforts and that the toolset facilitates many common tasks in OpenMP parallel programming in an efficient manner.

Parallel adaptive computing on meta-systems including NOWs

Load analysis of meta-systems including NOWs or COWs has shown that only a few percentage of the available power is used during long periods of time. Therefore, in order to exploit the idle time when executing a parallel application work load must be sent to a machine as soon as the latter becomes available. Furthermore, in order to keep respected the ownership of workstations work has to be stopped and resumed later as soon as the machine executing it is requisitioned by its owner. As a consequence, users need an adaptive system allowing to return events related to the goings and comings of workstations. On the other hand, it is necessary to provide them a parallel adaptive programming methodology that plans the handling of these events. In this paper, we present the MARS (MARS: multi-user adaptive resource scheduler, developed at LIFL laboratory, Universit é de Lille I) system and its parallel adaptive programming methodology through the block-based Gauss–Jordan algorithm used in numerical analysis to invert large matrices. Moreover, we propose a work scheduling strategy and an application-oriented solution for the fault tolerance issue. Furthermore, we present some experimental results obtained on a DEC/ALPHA COW and a SUN/Sparc4 NOW. The results show that very high absolute efficiencies can be obtained if the size of the blocks is well chosen. We also present some experimentations related to the adaptability of the application in a meta-system including the DEC/ALPHA COW and the SUN/Sparc4 NOW. The results show that the management of the adaptability consumes just a short percentage of execution time.

Selected parallel optimization methods for financial management under uncertainty

A review of some of the most important existing parallel solution algorithms for stochastic dynamic problems arising in financial planning is the main focus of this work. Optimization remains the most difficult, time and resource consuming part of the process of decision support for financial planning under uncertainty. However, other parts of a specialized decision support system (DSS) are also briefly outlined to provide appropriate background. Finally, financial modeling is but one of the possible application fields of stochastic dynamic optimization. Therefore the same fairly general methods described here are also useful in many other contexts. Authors hope that the overview of this application field may be of interest to readers concerned with development of parallel programming paradigms, methodology and tools. Therefore special care was taken to ensure that the presentation is easily understandable without much previous knowledge of theory and methods of operations research.

Structured Development of Parallel Programs

Susanna Pelagatti Taylor & Francis, Levittown, 1998, 192 pp. ISBN 0-7484-0759-6, $44.95 The main goal of this book is to propose, describe, and discuss a structured way for programming parallel computing architectures. As structured sequential programming improved the design and development of standard software on conventional computers, a structured approach is necessary to produce portable standard applications on parallel computers. This is the basic idea of the Pelagatti's book that deals with all the main topics related to programming parallel algorithms using a high-level programming methodology. Issues such as abstract models, programming constructs, mapping and scheduling techniques, compiling methods and performance models are discussed in the chapters of this text. The book can be used by advanced undergraduate and postgraduate students, researchers, and parallel programming professionals to learn a methodology for high-level parallel programming and understand the benefits that a structured approach to parallel programming may bring to the development of portable parallel software. It gives the reader a formal and complete view of a very promising field of parallel computation that may have a significant role in the next future of high-performance computing. To describe the proposed methodology based on structured parallel programming, the author divided the book into two main parts. The first part of the book, which consists of four chapters, introduces the basic issues of parallel programming and gives a survey of models and languages proposed in the literature. In particular, chapter 2 discusses the main problems to be solved when a parallel program must be designed and surveys the spectrum of the approaches (models, languages, and tools) to parallel software development proposed in the latest years. In chapter 3 the basic paradigms that are generally used for parallel computation are presented and their combination for building more complex techniques is discussed. These basic paradigms are used in the second part of the book for defining a structured methodology to parallel computation. The last two chapters of the first part of the book discuss program mapping and process scheduling issues and template-based systems that are strongly related to the structured methodology proposed in the book. The second part of the book discuss in detail a high-level methodology for parallel programs development based on skeletons as building blocks for parallel programming. In particular, a skeleton-based parallel language named P3L (Pisa Parallel Programming Language) developed at University of Pisa is extensively described. Skeletons are a set of basic forms of parallelism, such as pipeline, tree, farm, and loop, that embody both data- and control-parallelism. In chapters 6-9, the author gives a formal definition of skeletons, a clear description of the P3L constructs and an overview of the P3L compiler that generates an optimized implementation of skeleton-based programs on different classes of target parallel architectures. Finally, chapter 10 describes practical development of programs by P3L using some examples. Parallel computing machines are more and more available, but programming applications on these architectures is still difficult. The book represents a useful text for people wishing to study, design, and implement high-level parallel programs. It shows how important is the use of a high-level structured methodology in the development of parallel applications. Main issues such as abstractness, programmability, portability, and performance are addressed in a very accurate way in the text. The only minor remark that could be done is about the use of very few practical examples discussed in the second part of the book. However, overall, this text would be a valuable source to students and researchers studying and working on parallel computation. Domenico Talia, ISI-CNR, Rende, Italy

Parallel codes for computing the numerical rank

In this paper we present an experimental comparison of several numerical tools for computing the numerical rank of dense matrices. The study includes the well-known SVD, the URV decomposition, and several rank-revealing QR factorizations: the QR factorization with column pivoting, and two QR factorizations with restricted column pivoting. Two different parallel programming methodologies are analyzed in our paper. First, we present block-partitioned algorithms for the URV decomposition and rank-revealing QR factorizations which provide efficient implementations on shared memory environments. Furthermore, we also present parallel distributed algorithms, based on the message-passing paradigm, for computing rank-revealing QR factorizations on multicomputers.

GOFFIN: Higher-order functions meet concurrent constraints

We introduce a higher-order constraint-based language for structured and declarative parallel programming. The language, called Goffin, systematically integrates constraints and user-defined functions within a uniform setting of concurrent programming. From the perspective of parallel programming methodology, the constraint part of Goffin provides a co-ordination language for the functional part, which takes on the role of the computation language. This conceptual distinction allows the structured formulation of parallel algorithms. Goffin is an extension of the purely functional language Haskell. The functional kernel is embedded in a layer based on concurrent constraints. Logical variables are bound by constraints which impose relations over expressions that may contain user-defined functions. Referential transparency is preserved by restricting the creation of logical variables to the constraint part and by suspending the reduction of functional expressions that depend on the value of an unbound logical variable. Hence, constraints are the means to organize the concurrent reduction of functional expressions. Moreover, constraint abstractions, i.e., functions over constraints, allow the definition of parameterized co-ordination forms. In correspondence with the higher-order nature of the functional part, abstractions in the constraint logic part are based on higher-order logic, leading to concise and modular specifications of behaviour. We introduce and explain Goffin together with its underlying programming methodology, and present a declarative as well as an operational semantics for the language. To formalize the semantics, we identify the essential core constructs of the language and characterize their declarative meaning by associating them with formulae of Church's simple theory of types. We also present a reduction system that captures the concurrent operational semantics of the core constructs. In the course of this paper, the soundness of this reduction system with respect to the declarative semantics is established.

Deriving structured parallel implementations for numerical methods

The numerical solution of differential equations is an important problem in the natural sciences and engineering. But the computational effort to find a solution with the desired accuracy is usually quite large. This suggests the use of powerful parallel machines which often use a distributed memory organization. In this article, we present a parallel programming methodology to derive structured parallel implementations of numerical methods that exhibit two levels of potential parallelism, a coarse-grain method parallelism and a medium grain parallelism on data or systems. The derivation process is subdivided into three stages: The first stage identifies the potential for parallelism in the numerical method, the second stage fixes the implementation decisions for a parallel program and the third stage derives the parallel implementation for a specific parallel machine. The derivation process is supported by a group-SPMD computational model that allows the prediction of runtimes for a specific parallel machine. This enables the programmer to test different alternatives and to implement only the most promising one. We give several examples for the derivation of parallel implementations and of the performance prediction. Experiments on an Intel iPSC/860 confirm the accuracy of the runtime predictions. The parallel programming methodology separates the software issues from the architectural details, enables the design of well-structured, reusable and portable software and supplies a formal basis for automatic support.

P3L: A structured high‐level parallel language, and its structured support

AbstractThe paper presents a parallel programming methodology that ensures easy programming, efficiency and portability of programs to different machines belonging to the class of the general‐purpose, distributed‐memory, MIMD architectures. The methodology is based on the definition of a new, high‐level, explicitly parallel language, called P3 L, and of a set of static tools that automatically adapt the program features for each target architecture.P3 L does not require programmers to specify process activations, the actual parallelism degree, scheduling, or interprocess communications, i.e. all those features that need to be adjusted to harness each specific target machine. Parallelism is, on the other hand, expressed in a structured and qualitative way, by hierarchical composition of a restricted set of language constructs, corresponding to those forms of parallelism that are frequently encountered in parallel applications, and that can be efficiently implemented.The efficient portability of P3 L applications is guaranteed by the compiler along with the novel structure of the support. The compiler automatically adapts the program features for each specific architecture, using the costs (in terms of performance) of the low‐level mechanisms exported by the architecture itself. In our methodology, these costs, along with other features of the architecture, are viewed through an abstract machine, whose interface is used by the compiler to produce the final object code.

Graph-based software construction for parallel message-passing programs

A programming language which supports the adequate specification of both parallel and sequential aspects of a program establishes the optimal basis for a parallel programming methodology. Parallel languages that are entirely based on textual representations are not the best choice for describing paralellism. The main drawbacks stem from the fact that the sequential order of textual representations hides the parallel structure of a program. We propose a new programming methodology and language (called Meander), integrating textual and graphical descriptions, which is much better suited for parallel programming. Parallel aspects are formulated by means of a specification graph which is annotated by sequential code. Program design, coding and visualization can then be performed in one single formalism which is suitable for all phases.

Object‐Oriented Support for Adaptive Methods on Paranel Machines

This article reports on experiments from our ongoing project whose goal is to develop a C++ library which supports adaptive and irregular data structures on distributed memory supercomputers. We demonstrate the use of our abstractions in implementing "tree codes" for large‐scale N‐body simulations. These algorithms require dynamically evolving treelike data structures, as well as load‐balancing, both of which are widely believed to make the application difficult and cumbersome to program for distributed‐memory machines. The ease of writing the application code on top of our C++ library abstractions (which themselves are application independent), and the low overhead of the resulting C++ code (over hand‐crafted C code) supports our belief that object‐oriented approaches are eminently suited to programming distributed‐memory machines in a manner that (to the applications programmer) is architecture‐independent. Our contribution in parallel programming methodology is to identify and encapsulate general classes of communication and load‐balancing strategies useful across applications and MIMD architectures. This article reports experimental results from simulations of half a million particles using multiple methods.