Parallel Execution Of Loops Research Articles

HPSM: a programming framework to exploit multi-CPU and multi-GPU systems simultaneously

This paper presents a high-level C++ framework to explore multi-CPU and multi-GPU systems called HPSM. HPSM enables execution of parallel loops and reductions simultaneously over CPUs and GPUs using three parallel backends: Serial, OpenMP, and StarPU. We analysed HPSM development effort with AXPY program through two standard metrics (NCLOC and ES). In addition, we evaluated performance and energy with three parallel benchmarks: N-Body, Hotspot, and CFD solver. HPSM reduced code effort by up to 56.9% compared to StarPU C interface, although it resulted in 2.5× more lines of code compared to OpenMP. The CPU-GPU combination attained speedup results with Hotspot of up to 92.7× on a X86-based system with four GPUs and up to 108.2× on an IBM POWER8+ system with two GPUs. On both systems, the addition of GPUs improved energy efficiency.

HPSM: a programming framework to exploit multi-CPU and multi-GPU systems simultaneously

This paper presents a high-level C++ framework to explore multi-CPU and multi-GPU systems called HPSM. HPSM enables execution of parallel loops and reductions simultaneously over CPUs and GPUs using three parallel backends: Serial, OpenMP, and StarPU. We analysed HPSM development effort with AXPY program through two standard metrics (NCLOC and ES). In addition, we evaluated performance and energy with three parallel benchmarks: N-Body, Hotspot, and CFD solver. HPSM reduced code effort by up to 56.9% compared to StarPU C interface, although it resulted in 2.5× more lines of code compared to OpenMP. The CPU-GPU combination attained speedup results with Hotspot of up to 92.7× on a X86-based system with four GPUs and up to 108.2× on an IBM POWER8+ system with two GPUs. On both systems, the addition of GPUs improved energy efficiency.

Fault Tolerant Scheduling for Parallel Loops on Shared Memory Systems

While multicore/multiprocessor systems achieve significant speedup for many applications by exploiting loop level parallelism, they also suffer from increased reliability problems as a result of ever scaling device size. This paper addresses the reliability of loop dominated applications, aiming to execute parallel loops efficiently in the presence of various types of hardware faults. In this paper, we present a fault tolerant work-stealing scheme which makes parallel loop execution resilient to hardware faults. A lightweight buffer-commit mechanism is applied in the proposed scheme to ensure the correctness of the re-execution of loop iterations. In addition, we split large failing chunks of loop iterations at runtime to improve load balancing, and a worker thread is discarded when faults occur frequently on it. We evaluated our techniques on a multi-socket multicore system, using a set of loop dominated benchmarks. The proposed scheme achieves the minimum overhead of supporting fault tolerance and optimal load balancing.

Performance Evaluation and Enhancement of Process-Based Parallel Loop Execution

Parallel programming is known to be difficult and error-prone. Thread-based parallel execution has particular difficulties due to the tendency for the program to contain errors such as incorrect operation ordering, atomicity violation, and others. Worse yet, as many of such erroneous behaviors tend to be non-deterministic, the programmer is often unable to reproduce the exact event sequence that causes the program failure, which makes diagnosis difficult. In contrast, with process-based parallel execution, unintended data sharing can be avoided, thanks to the isolated address spaces among processes, which greatly simplifies the run-time program states, making it easier to reproduce and diagnose an error. Nonetheless, parallel loop execution on multicore has been dominated by parallel threads and thread-based language extensions and tools. This seems to be due to a long-held common wisdom that process-based parallel execution incurs much higher overhead. This paper reports experimental results that show the competitiveness of process-based parallel loop execution. Several benchmark programs of process-based parallel execution achieved speedups ranging from 6.73 to 20.24 on a 32 cores machine.

Автоматизированное преобразование ФОРТРАН-программ, необходимое для их эффективного распараллеливания с помощью системы САПФОР

Automatic mapping of sequential programs on height performance computers with distributed memory may require preliminary transformation of programs oriented to this class of systems. Using the SAPFOR system for parallelization of applications allowed to explore transformations which can be executed in an automated way. The paper presents transformations that increase the possibility of efficient parallelization of programs by eliminating the reasons that prohibit the parallel execution of loops. Implementation of these transformations allowed to automate the obtainment of serial hydrodynamic program, which may be efficiently mapped on modern clusters by the automatically parallelizing compiler included in the SAPFOR.

Guest Editorial: Application Specific Processors and Architectures

Application specific processors and architectures are becoming increasingly important across all fields of computing from embedded to high-performance systems. These architectures and systems require efficient arithmetic algorithms, compilers, operating systems, and specialized applications to achieve optimum performance under strict constraints including size, power, throughput and operating frequency. This special double issue focuses on the latest developments and solutions concerning application specific processors and architectures from the hardware and software perspectives. This special issue consists of eleven papers related to the area of application-specific processors and architectures that are divided into three categories: compilers and operating systems, arithmetic algorithms, and application/algorithm specialization. The first paper in the compilers and operating systems category “Compact Code Generation for TightlyCoupled Processor Arrays”, by Boppu et al., presents methods for code compaction and generation for programmable tightly-coupled processor arrays consisting of interconnected small light-weight VLIW cores. The authors integrate these methods into a design tool and evaluate the results with benchmarks and compare the results to other existing compiler frameworks. The methods exploit compute-intensive nested loops, providing design entry in the form of a functional programming language and loop parallelization in the polyhedron model. The methods also support zero-overhead looping not only for the innermost loops but also for arbitrarily nested loops. The next paper, “Symbolic Mapping of Loop Programs onto Processor Arrays”, by Teich, et al., presents a solution to the problem of joint tiling and scheduling of a loop nest with uniform data dependencies symbolically. This challenge arises when the size and number of available processors for parallel loop execution is not known at compile time. The paper discusses a solution for deriving parameterized latency-optimal schedules statically by proposing a two step approach that determines two potential schedules. Once the size of the processor array becomes known at run time, simple comparisons of latency-determining expressions finally steer which of these schedules will be dynamically selected and the corresponding program configuration executed on the resulting processor array so to avoid any further run-time optimization or expensive recompilation. In the paper “Virtualized Execution and Management of Hardware Tasks on a Hybrid ARM-FPGA Platform”, by Jain et al., the authors focus on managing the execution of hardware tasks within a processor-based system, and in doing so, how to virtualize the resources to ensure isolation and predictability. The authors use a microkernel-based hypervisor running on a commercial hybrid (FPGA-based) computing platform. The hypervisor leverages the capabilities of the FPGA fabric, with support for discrete hardware accelerators, dynamically reconfigurable regions, and regions of virtual fabric. The authors study the communication overheads, quantify the context switch overhead of the hypervisor approach and compare with the idle time for a standard Linux implementation, showing two orders of magnitude improvement. The final compilers and operating systems paper, “ANovel Object-Oriented Software Cache for Scratchpad-BasedMultiCore Clusters”, by Pinto et al., presents a software cache implementation for an accelerator fabric, with special focus on object-oriented caching techniques aimed at reducing the global overhead introduced by the proposed software cache. The authors validate the software cache approach with a set of experiments, and three case studies of Object-Oriented software cache for computer vision applications. There are four papers related to application implementations in this special issue. The first paper, “Hardware Acceleration of M. C. Smith (*) Holcombe, Department of Electrical and Computer Engineering, Clemson University, Clemson, SC, USA e-mail: smithmc@clemson.edu

Symbolic Mapping of Loop Programs onto Processor Arrays

In this paper, we present a solution to the problem of joint tiling and scheduling a given loop nest with uniform data dependencies symbolically. This challenge arises when the size and number of available processors for parallel loop execution is not known at compile time. But still, in order to avoid any overhead of dynamic (run-time) recompilation, a schedule of loop iterations shall be computed and optimized statically. In this paper, it will be shown that it is possible to derive parameterized latency-optimal schedules statically by proposing a two step approach: First, the iteration space of a loop program is tiled symbolically into orthotopes of parametrized extensions. Subsequently, the resulting tiled program is also scheduled symbolically, resulting in a set of latency-optimal parameterized schedule candidates. At run time, once the size of the processor array becomes known, simple comparisons of latency-determining expressions finally steer which of these schedules will be dynamically selected and the corresponding program configuration executed on the resulting processor array so to avoid any further run-time optimization or expensive recompilation. Our theory of symbolic loop parallelization is applied to a number of loop programs from the domains of signal processing and linear algebra. Finally, as a proof of concept, we demonstrate our proposed methodology for a massively parallel processor array architecture called tightly coupled processor array (TCPA) on which applications may dynamically claim regions of processors in the context of invasive computing.

ATLS - A parallel loop scheduling scheme for dynamic environments

We here present ATLS, a self scheduling scheme designed for execution of parallel loops in dynamic environments of non-dedicated networked computers. Since grid and volunteer systems based on desktop computers are proving themselves as real and powerful alternatives for parallel computing, new scheduling schemes, better suited for these environments, are required. Our proposal, by tracking several performance change ratios at runtime, is able to properly adjust the load distribution using no prior information of the loop features nor the involved processors. The results obtained during the experiments performed to validate ATLS show that it is possible to improve former contributions of well-known parallel loop scheduling schemes in dynamic environments. The implementation of the scheduler has been done for the R language but, as it is exposed, it can easily be adapted to any other language and parallel loop based application.

R/parallel – speeding up bioinformatics analysis with R

BackgroundR is the preferred tool for statistical analysis of many bioinformaticians due in part to the increasing number of freely available analytical methods. Such methods can be quickly reused and adapted to each particular experiment. However, in experiments where large amounts of data are generated, for example using high-throughput screening devices, the processing time required to analyze data is often quite long. A solution to reduce the processing time is the use of parallel computing technologies. Because R does not support parallel computations, several tools have been developed to enable such technologies. However, these tools require multiple modications to the way R programs are usually written or run. Although these tools can finally speed up the calculations, the time, skills and additional resources required to use them are an obstacle for most bioinformaticians.ResultsWe have designed and implemented an R add-on package, R/parallel, that extends R by adding user-friendly parallel computing capabilities. With R/parallel any bioinformatician can now easily automate the parallel execution of loops and benefit from the multicore processor power of today's desktop computers. Using a single and simple function, R/parallel can be integrated directly with other existing R packages. With no need to change the implemented algorithms, the processing time can be approximately reduced N-fold, N being the number of available processor cores.ConclusionR/parallel saves bioinformaticians time in their daily tasks of analyzing experimental data. It achieves this objective on two fronts: first, by reducing development time of parallel programs by avoiding reimplementation of existing methods and second, by reducing processing time by speeding up computations on current desktop computers. Future work is focused on extending the envelope of R/parallel by interconnecting and aggregating the power of several computers, both existing office computers and computing clusters.

Possibilities of Optimal Execution of Parallel Programs Containing Simple and Iterated Loops on Heterogeneous Parallel Computational Systems with Distributed Memory

The problem of load balancing when executing parallel programs on computational systems with distributed memory is currently of great interest. The most general statement of this problem is that for one parallel loop: execution of a heterogeneous loop on a heterogeneous computational system. When stated in this way, the problem is NP-complete even in the case of two nodes, and no acceptable heuristics for solving it are found. Since the development of heuristics is a rather complicated task, we decided to examine the problem by elementary methods in order to refine (and, possibly, simplify) the original problem statement. The results of our studies are discussed in this paper. Estimates of efficiency of parallel loop execution as functions of the number of nodes of homogeneous and heterogeneous parallel computational systems are obtained. These estimates show that the use of heterogeneous parallel systems reduces the efficiency even in the case when their communication subsystems are scaleable (see the definition in Section 4). The use of local networks (heterogeneous parallel computational systems with nonscaleable communication subsystems) for parallel computations with heavy data exchange is not advantageous and is possible only for a small number of nodes (about five). An algorithm of optimal distribution of data between the nodes of a homogeneous or heterogeneous computational system is suggested. Results of numerical experiments substantiate the conclusions obtained.

A two-stage solution approach to multidimensional periodic scheduling

We present a two-stage solution approach to the multidimensional periodic scheduling (MPS) problem. This problem originates from the design of high-throughput digital-signal-processor systems, where highly parallel execution of loops is of utmost importance. We introduce the concept of multidimensional periodic operations in order to cope with problems originating from loop hierarchies and explicit timing requirements. In the first stage of the approach, we assign periods to the multidimensional periodic operations such that storage costs are minimized. This is done by means of branch-and-bound, based on a linear programming and constraint-generation technique. In the second stage, we assign start times to the operations and determine on which processing units (PUs) they are executed. This is done by means of an iterative approach. The two major subproblems of MPS concerning checking data dependency constraints and PU constraints are solved by means of an all-integer integer-linear-programming technique. This technique is used as a subroutine in the above two stages. The effectiveness and efficiency of the approach are good, which is illustrated by means of some practical examples.

SUPPLE: An efficient run-time support for non-uniform parallel loops

Abstract This paper presents SUPPLE (SUPort for Parallel Loop Execution), an innovative run-time support for the execution of parallel loops with regular stencil data references and non-uniform iteration costs. SUPPLE relies upon a static block data distribution to exploit locality, and combines static and dynamic policies for scheduling non-uniform iterations. It adopts, as far as possible, a static scheduling policy derived from the owner computes rule, and moves data and iterations among processors only if a load imbalance actually occurs. SUPPLE always tries to overlap communications with useful computations by reordering loop iterations and prefetching remote ones in the case of workload imbalance. The SUPPLE approach has been validated by many experimental results obtained by running a multi-dimensional flame simulation kernel on a 64-node Cray T3D. We have fed the benchmark code with several synthetic input data sets built on the basis of a load imbalance model. We have compared our results with those obtained with a CRAFT Fortran implementation of the benchmark.

The impact of parallel loop scheduling strategies on prefetching in a shared memory multiprocessor

Trace-driven simulations of numerical Fortran programs are used to study the impact of the parallel loop scheduling strategy on data prefetching in a shared memory multiprocessor with private data caches. The simulations indicate that to maximize memory performance, it is important to schedule blocks of consecutive iterations to execute on each processor, and then to adaptively prefetch single-word cache blocks to match the number of iterations scheduled. Prefetching multiple single-word cache blocks on a miss reduces the miss ratio by approximately 5% to 30% compared to a system with no prefetching. In addition, the proposed adaptive prefetching scheme further reduces the miss ratio while significantly reducing the false sharing among cache blocks compared to nonadaptive prefetching strategies. Reducing the false sharing causes fewer coherence invalidations to be generated, and thereby reduces the total network traffic. The impact of the prefetching and scheduling strategies on the temporal distribution of coherence invalidations also is examined. It is found that invalidations tend to be evenly distributed throughout the execution of parallel loops, but tend to be clustered when executing sequential program sections. The distribution of invalidations in both types of program sections is relatively insensitive to the prefetching and scheduling strategy.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Overlapped loop support in the Cydra 5

The Cydra TM 5 architecture adds unique support for overlapping successive iterations of a loop to a very long instruction word (VLIW) base. This architecture allows highly parallel loop execution for a much larger class of loops than can be vectorized, without requiring the unrolling of loops usually used by compilers for VLIW machines. This paper discusses the Cydra 5 loop scheduling model, the special architectural features which support it, and the loop compilation techniques used to take full advantage of the architecture.

Parallel execution of loops: The pyramid method

This article proposes a method of isolating independent program branches based on finding the resulting iterations and then solving the problem of minimizing the execution time for a given number of processors or minimizing the number of processors for a fixed execution time. The distribution of iterations can be organized in a more regular pattern given the interconnection through variables with linear indices of a certain simple form. Each family of iterations connected informationally can be seen as a pyramid in the space of iterations. Paralleling in programs involves repeating areas of calculation by using loops. An attempt is made to solve the problem of paralleling with complete or at least maximum possible independence in the execution of parallel branches, even if it involves the duplication of certain computation fragments in different branches.

The parallel execution of loops: The parallelepiped method

The parallel execution of loops: The parallelepiped method