Performance Of Parallel Programs Research Articles

Disentanglement with Futures, State, and Interaction

Recent work has proposed a memory property for parallel programs, called disentanglement, and showed that it is pervasive in a variety of programs, written in different languages, ranging from C/C++ to Parallel ML, and showed that it can be exploited to improve the performance of parallel functional programs. All existing work on disentanglement, however, considers the "fork/join" model for parallelism and does not apply to "futures", the more powerful approach to parallelism. This is not surprising: fork/join parallel programs exhibit a reasonably strict dependency structure (e.g., series-parallel DAGs), which disentanglement exploits. In contrast, with futures, parallel computations become first-class values of the language, and thus can be created, and passed between functions calls or stored in memory, just like other ordinary values, resulting in complex dependency structures, especially in the presence of mutable state. For example, parallel programs with futures can have deadlocks, which is impossible with fork-join parallelism. In this paper, we are interested in the theoretical question of whether disentanglement may be extended beyond fork/join parallelism, and specifically to futures. We consider a functional language with futures, Input/Output (I/O), and mutable state (references) and show that a broad range of programs written in this language are disentangled. We start by formalizing disentanglement for futures and proving that purely functional programs written in this language are disentangled. We then generalize this result in three directions. First, we consider state (effects) and prove that stateful programs are disentangled if they are race free. Second, we show that race freedom is sufficient but not a necessary condition and non-deterministic programs, e.g. those that use atomic read-modify-operations and some non-deterministic combinators, may also be disentangled. Third, we prove that disentangled task-parallel programs written with futures are free of deadlocks, which arise due to interactions between state and the rich dependencies that can be expressed with futures. Taken together, these results show that disentanglement generalizes to parallel programs with futures and, thus, the benefits of disentanglement may go well beyond fork-join parallelism.

The Role of Idle Waves, Desynchronization, and Bottleneck Evasion in the Performance of Parallel Programs

The performance of highly parallel applications on distributed-memory systems is influenced by many factors. Analytic performance modeling techniques aim to provide insight into performance limitations and are often the starting point of optimization efforts. However, coupling analytic models across the system hierarchy (socket, node, network) fails to encompass the intricate interplay between the program code and the hardware, especially when execution and communication bottlenecks are involved. In this paper we investigate the effect of "bottleneck evasion" and how it can lead to automatic overlap of communication overhead with computation. Bottleneck evasion leads to a gradual loss of the initial bulk-synchronous behavior of a parallel code so that its processes become desynchronized. This occurs most prominently in memory-bound programs, which is why we choose memory-bound benchmark and application codes, specifically an MPI-augmented STREAM Triad, sparse matrix-vector multiplication, and a collective-avoiding Chebyshev filter diagonalization code to demonstrate the consequences of desynchronization on two different supercomputing platforms. We investigate the role of idle waves as possible triggers for desynchronization and show the impact of automatic asynchronous communication for a spectrum of code properties and parameters, such as saturation point, matrix structures, domain decomposition, and communication concurrency. Our findings reveal how eliminating synchronization points (such as collective communication or barriers) precipitates performance improvements that go beyond what can be expected by simply subtracting the overhead of the collective from the overall runtime.

Parallel optimization over the integer efficient set

AbstractThis paper introduces a modified sequential version method for optimizing a linear function over an integer efficient set, as well as a new exact parallel algorithm. The performance of parallel programming in this context is clear and shown through different instances with different sizes. Each procedure builds a finite monotonous sequence of values for the main criterion to be optimized, in a reasonable amount of CPU execution time. This latter remains much better. For the first time, the Algerian IBNBADIS cluster—CERIST—was used with this type of problem. Significant results are obtained by both proposed techniques, particularly with the parallel one.

Critical Section Overhead Reduction for OpenMP Program by Nesting a Serial Loop to Increase Task Granularity of Parallel Loop

This paper presents a simple method to reduce performance loss due to a parallel program's massive critical sections of parallel numerical integration. The method is to transform a fine grain parallel loop into a coarse grain parallel loop which is nesting a sequential loop. The coarse grain parallel loop is by nesting a loop block to make task granularities coarser than that naive one. In addition to the overhead reduction, the method makes the parallel work fraction significantly larger than the serial fraction. As a result, nesting a serial loop within a parallel loop improves the parallel program’s performance. Compared to the naïve method, which does not scale performance of parallel program of numerical integration, the nesting serial loop method scales a parallel program up to 3.26 times fold relative to its sequential program on quad-core processor. This result shows that the proposed method makes parallel program much faster compared to the naïve method.

Multicore Performance Prediction with MPET

Multicore processors serve as target platforms in a broad variety of applications ranging from high-performance computing to embedded mobile computing and automotive applications. But, the required parallel programming opens up a huge design space of parallelization strategies each with potential bottlenecks. Therefore, an early estimation of an application’s performance is a desirable development tool. However, out-of-order execution, superscalar instruction pipelines, as well as communication costs and (shared-) cache effects essentially influence the performance of parallel programs. While offering low modeling effort and good simulation speed, current approximate analytic models provide moderate prediction results so far. Virtual prototyping requires a time-consuming simulation, but produces better accuracy. Furthermore, even existing statistical methods often require detailed knowledge of the hardware for characterization. In this work, we present a concept called Multicore Performance Evaluation Tool (MPET) and its evaluation for a statistical approach for performance prediction based on abstract runtime parameters, which describe an application’s scalability behavior and can be extracted from profiles without user input. These scalability parameters not only include information on the interference of software demands and hardware capabilities, but indicate bottlenecks as well. Depending on the database setup, we achieve a competitive accuracy of 20% mean prediction error (11% median), which we also demonstrate in a case study.

Automated Performance Modeling of HPC Applications Using Machine Learning

Automated performance modeling and performance prediction of parallel programs are highly valuable in many use cases, such as in guiding task management and job scheduling, offering insights of application behaviors, and assisting resource requirement estimation. The performance of parallel programs is affected by numerous factors, including but not limited to hardware, applications, algorithms, and input parameters, thus an accurate performance prediction is often a challenging and daunting task. In this article, we focus on automatically predicting the execution time of parallel programs (more specifically, MPI programs) with different inputs, at different scales, and without domain knowledge. We model the correlation between the execution time and domain-independent runtime features. These features include values of variables, counters of branches, loops, and MPI communications. Through automatically instrumenting an MPI program, each execution of the program will output a feature vector and its corresponding execution time. After collecting data from executions with different inputs, a random forest machine learning approach is used to build an empirical performance model, which can predict the execution time of the program given a new input. A transfer learning method is used to reuse an existing performance model and improve the prediction accuracy on a new platform that lacks historical execution data. Our experiments and analyses of three parallel applications, Graph500, GalaxSee, and SMG2000, on three different systems confirm that our method performs well, with less than 20 percent prediction error on average.

Performance Modeling of Parallel Loops on Multi-Socket Platforms Using Queueing Systems

Predicting the performance of parallel loops on modern shared-memory multi-socket multi-core systems in dependence of the allocated resources is an important means to achieve better system utilization. Previous prediction techniques are tied to specific architectures and do not allow for purely online performance predictions without requiring an offline analysis of the parallel program. This paper presents a practical approach based on queueing theory to model the performance of parallel programs in dependence of the number of allocated core resources. Based on the key insight that scalability of scientific parallel loops is limited by memory performance, a hierarchically constructed M/M/1/N/N queue system is used to analytically compute the response time at the different congestion points in the memory system of modern NUMA architectures. After automatically tuning the model to a specific architecture by executing a number of micro-benchmarks, the required parameter values are obtained at runtime from hardware performance counters present in modern commodity AMD and Intel processors. Evaluated with 24 OpenMP parallel loops on a 64-core AMD and a 72-core Intel multi-socket platform, the presented queueing system is able to accurately predict the speedup of parallel loops with a mean absolute percentage error of 8.3 percent on the AMD system and 6.7 percent on the Intel platform.

Visualizing Hierarchical Performance Profiles of Parallel Codes Using CallFlow.

Calling context trees (CCTs) couple performance metrics with call paths, helping understand the execution and performance of parallel programs. To identify performance bottlenecks, programmers and performance analysts visually explore CCTs to form and validate hypotheses regarding degraded performance. However, due to the complexity of parallel programs, existing visual representations do not scale to applications running on a large number of processors. We present CallFlow, an interactive visual analysis tool that provides a high-level overview of CCTs together with semantic refinement operations to progressively explore CCTs. Using a flow-based metaphor, we visualize a CCT by treating execution time as a resource spent during the call chain, and demonstrate the effectiveness of our design with case studies on large-scale, production simulation codes.

Enhancing transactional memory execution via dynamic binary translation

Transactional Synchronization Extensions (TSX) have been introduced for hardware transactional memory since the 4th generation Intel Core processors. TSX provides two software programming interfaces: Hardware Lock Elision (HLE) and Restricted Transactional Memory (RTM). HLE is easy to use and maintains backward compatibility with processors without TSX support, while RTM is more flexible and scalable. Previous researches have shown that critical sections protected by RTM with a well-designed retry mechanism as its fallback code path can often achieve better performance than HLE. More parallel programs may be programmed in HLE, however, using RTM may obtain greater performance. To embrace both productivity and high performance of parallel programs with TSX, we present a framework built on QEMU that can dynamically transform HLE instructions in an application binary to fragments of RTM codes with adaptive tuning on the fly. Compared to HLE execution, our prototype achieves 1.56 x speedup with 8 threads on average. Due to the scalability of RTM, the speedup will be more significant as the number of threads increases.

Overcoming the No Free Lunch Theorem in Cut-off Algorithms for Fork-Join programs

Abstract The No Free Lunch Theorem establishes that for any algorithm, any elevated performance over one class of problems is offset by performance over another class. Empirically, this is true for Granularity Control algorithms, particularly in Fork-Join style concurrent programs. Cut-off Algorithms make the decision to either continue forking work (parallelising) or instead computing the remaining execution branches sequentially. These algorithms affect how well a program dynamically adjusts to the parallelism available in the machine. To put things into context, choosing the wrong algorithm can make a program go from 30 s of execution to over a day. Cut-off Algorithms such as Max-Tasks, Max-Level, SurplusTasks and variants do not provide the same performance improvements over the same classes of Fork-Join programs. Thus, programmers have to decide which Cut-off Algorithms to use on different classes of problems. Finding the right Cut-off Algorithm is a time-consuming task, as, most of times, it requires trial-and-error execution of several configurations before determining the right solution. To the best of our knowledge, there is no definitive solution for making this decision automatically. Being able to do so would largely enhance the performance of parallel programs generated by optimising compilers and/or executed by parallelising frameworks. In this work, we address the problem of automatically selecting the right Cut-off Algorithm for highly parallel programs. We evaluate the performance of nowadays most relevant Cut-off Algorithms over different programs. We identify possible correlations between program characteristics and performance improvements observed with different cut-off strategies. Finally, we derive a set classification rules for selection of Cut-off Algorithms that can be integrated into compilers and frameworks that perform automatic parallelisation of programs.

The electromagnetic waves propagation in unmagnetized plasma media using parallelized finite-difference time-domain method

The finite-difference time-domain (FDTD) method has been commonly utilized to simulate the electromagnetic (EM) waves propagation in the plasma media. However, the FDTD method may bring about extra run-time on concerning computationally large and complicated EM problems. Fortunately, the FDTD method is easy to parallelize. Besides, GPU has been widely used for parallel computing due to its unique SPMD (Single Program Multiple Data) architecture. In this paper, we represent the parallel Runge-Kutta exponential time differencing scheme FDTD (RKETD) method for the unmagnetized plasma implemented on GPU. The detailed flowchart of parallel RKETD-FDTD method is described. The accuracy and acceleration performance of the proposed parallel RKETD-FDTD method implemented on GPU are substantiated by calculating the reflection and transmission coefficients for one-dimensional unmagnetized plasma slab. The results indicate that the numerical precision of the parallel RKETD-FDTD scheme is consistent with that of the code implemented on CPU. The computation efficiency is greatly improved compared with merely CPU-based serial RKETD-FDTD method. Moreover, the comparisons of the performance of CUDA-based GPU parallel program, Open MP (Open Multi-Processing)-based CPU parallel program, and single-CPU serial program on the same host computer are done. Compared with the serial program, both parallel programs get good results, while GPU-based parallel program gains better result.

Let’s HPC: A web-based platform to aid parallel, distributed and high performance computing education

Let’s HPC (www.letshpc.org) is an evolving open-access web-based platform to supplement conventional classroom oriented High Performance Computing (HPC) and Parallel & Distributed Computing (PDC) education. This platform has been developed to allow users to learn, evaluate, teach and see the performance of parallel algorithms from a system’s viewpoint. The Let’s HPC platform’s motivation comes from the experiences of teaching HPC/PDC courses and it is designed to help streamline the process of analyzing parallel programs.At the heart of this platform is a database archiving the performance and execution environment related data of standard parallel algorithm implementations run on different computing architectures using different programming environments. The online plotting and analysis tools of our platform can be combined seamlessly with the database to aid self-learning, teaching, evaluation and discussion of different HPC related topics, with a particular focus on a holistic system’s perspective. The user can quantitatively compare and understand the importance of numerous deterministic as well as non-deterministic factors of both the software and the hardware that impact the performance of parallel programs. Instructors of HPC/PDC related courses can use the platform’s tools to illustrate the importance of proper data collection and analysis in understanding factors impacting performance as well as to encourage peer learning among students. Scripts are provided for automatically collecting performance related data, which can then be analyzed using the platform’s tools. The platform also allows students to prepare a standard lab/project report aiding the instructor in uniform evaluation. The platform’s modular design enables easy inclusion of performance related data from contributors as well as addition of new features in the future.This paper summarizes the background and motivation behind the Let’s HPC project, the design philosophy of the platform, the present capabilities of the platform, as well as the plans for future developments.

OpenMP performance for benchmark 2D shallow water equations using LBM

Shallow water equations or commonly referred as Saint-Venant equations are used to model fluid phenomena. These equations can be solved numerically using several methods, like Lattice Boltzmann method (LBM), SIMPLE-like Method, Finite Difference Method, Godunov-type Method, and Finite Volume Method. In this paper, the shallow water equation will be approximated using LBM or known as LABSWE and will be simulated in performance of parallel programming using OpenMP. To evaluate the performance between 2 and 4 threads parallel algorithm, ten various number of grids Lx and Ly are elaborated. The results show that using OpenMP platform, the computational time for solving LABSWE can be decreased. For instance using grid sizes 1000 × 500, the speedup of 2 and 4 threads is observed 93.54 s and 333.243 s respectively.

Parallelization of elliptic solver for solving 1D Boussinesq model

In this paper, a parallel implementation of an elliptic solver in solving 1D Boussinesq model is presented. Numerical solution of Boussinesq model is obtained by implementing a staggered grid scheme to continuity, momentum, and elliptic equation of Boussinesq model. Tridiagonal system emerging from numerical scheme of elliptic equation is solved by cyclic reduction algorithm. The parallel implementation of cyclic reduction is executed on multicore processors with shared memory architectures using OpenMP. To measure the performance of parallel program, large number of grids is varied from 28 to 214. Two test cases of numerical experiment, i.e. propagation of solitary and standing wave, are proposed to evaluate the parallel program. The numerical results are verified with analytical solution of solitary and standing wave. The best speedup of solitary and standing wave test cases is about 2.07 with 214 of grids and 1.86 with 213 of grids, respectively, which are executed by using 8 threads. Moreover, the best efficiency of parallel program is 76.2% and 73.5% for solitary and standing wave test cases, respectively.

Performance challenges in modular parallel programs

Over the past decade, many programming languages and systems for parallel-computing have been developed, including Cilk, Fork/Join Java, Habanero Java, Parallel Haskell, Parallel ML, and X10. Although these systems raise the level of abstraction at which parallel code are written, performance continues to require the programmer to perform extensive optimizations and tuning, often by taking various architectural details into account. One such key optimization is granularity control, which requires the programmer to determine when and how parallel tasks should be sequentialized. In this paper, we briefly describe some of the challenges associated with automatic granularity control when trying to achieve portable performance for parallel programs with arbitrary nesting of parallel constructs. We consider a result from the functional-programming community, whose starting point is to consider an "oracle" that can predict the work of parallel codes, and thereby control granularity. We discuss the challenges in implementing such an oracle and proving that it has the desired theoretical properties under the nested-parallel programming model.

PRODA: improving parallel programs on GPUs through dependency analysis

GPU’s powerful parallel processing capability has been highly recognized throughout the industry; however, GPU computing environments have not yet been widely used in the field of parallel computing. In this study, we develop a method of parallelization of serial programs for GPU computing. In particular, we propose an approach called PRODA to speedup parallel programs on GPUs through dependency analysis. PRODA provides theoretical underpins of task partitioning in parallel programs running in GPU computing environments. At the heart of PRODA is an analyzer for program workflows as well as data and function dependencies in a GPU program. With the dependency analysis in place, PRODA assigns computing tasks to multiple GPU cores in a way to speedup the performance of parallel program on GPUs. An overarching goal of PRODA is to minimize data communication cost between GPUs and main memory of a host CPU. PRODA achieves this goal by apply deploying two strategies. First, PRODA assigns functions processing the same data to a GPU core. Second, PRODA runs multiple independent functions on separate GPU cores. In doing so, PRODA improves the parallelism of parallel programs. We evaluate the performance of PRODA by running two popular benchmarks (i.e., AES and T26) on an 256-core system, where key length is set to 256 bits. The experimental results show that the speedup ratio of AES governed by PRODA is 5.2. Specifically, PRODA improves the performance of the existing CFM scheme by a factor of 1.39. To measure cost of parallel computing, we test PRODA and the alternative solutions by running AES under the 256-bit key length on 128 cores. The cost of parallel computing in PRODA is 524.8ms, which is 61.2% lower than that of the existing SA solution. The parallel efficiency of PRODA is 2.08, which represents an improvement of the PDM algorithm by a factor of 2.08.

Understanding the impact of task granularity in the energy consumption of parallel programs

Recently, there is a concern about reducing the energy consumption of data centers and clusters for economical and environmental reasons. Furthermore, energy consumption on mobile devices is also important to improve battery life. In this work we address the performance-energy trade-off on shared-memory multicore devices in parallel programs. In particular, we assess the impact of task granularity in performance and energy consumption. Our aim is to give programmers the knowledge they need to understand how to maximize performance of parallel programs while minimizing energy spending.Parallel programs typically divide work in subproblems that are solved in parallel. Each subproblem can then be recursively subdivided until it is no longer worthwhile to spawn smaller tasks. Ideally, the number of parallel tasks should match the number of hardware threads in order to maximize performance and reduce scheduling overheads. Cut-off algorithms are used to stop spawning new parallel tasks and, thus, switching to sequential execution. We evaluate cut-off approaches such as MaxTasks, MaxLevel, Surplus, Adaptive Task Control and LoadBased to understand how they influence performance and energy consumption. Additionally, we also introduce and evaluate three novel approaches: MaxTasksInQueue, StackSize and MaxTasksWithStackSize.Our experiments and analysis show how branching, workload, depth and balance influence the execution time and energy spending over a set of synthetic and real world programs. We concluded that MaxLevel was the fastest overall, while MaxTasksInQueue was the most energy efficient algorithm. Also, despite MaxTasks being slower than the prior two, it can be used by a wider range of programs.

Predicting HPC parallel program performance based on LLVM compiler

Performance prediction of parallel program plays key roles in many areas, such as parallel system design, parallel program optimization, and parallel system procurement. Accurate and efficient performance prediction on large-scale parallel systems is a challenging problem. To solve this problem, we present an effective framework for performance prediction based on the LLVM compiler technique in this paper. We can predict the performance of a parallel program on a small amount of nodes of the target parallel system using this framework toned but not execute this parallel program on a corresponding full-scale parallel system. This framework predicts the performance of computation and communication components separately and combines the two predictions to achieve full program prediction. As for sequential computation, we first combine the static branch probability and loop trip count identification and propose a new instrumentation method to acquire the number of each instruction type. We then construct a test program to measure the average execution time of each instruction type. Finally, we utilize the pruning technique to convert a parallel program into a corresponding sequential program to predict the performance on only one node of the target parallel system. As for communication, we utilize the LogGP model to model point-to-point communication and the artificial neural network technique to model collective communication. We validate our approach by a set of experiments that predict the performance of NAS parallel benchmarks and CGPOP parallel application. Experimental results show that the proposed framework can accurately predict the execution time of parallel programs, and the average error rate of these programs is 10.86%.

Understanding Parallel Performance Under Interferences in Multi-tenant Clouds

The performance of parallel programs is notoriously difficult to reason in virtualized environments. Although performance degradations caused by virtualization and interferences have been well studied, there is little understanding why different parallel programs have unpredictable slow- downs. We find that unpredictable performance is the result of complex interplays between the design of the program, the memory hierarchy of the hosting system, and the CPU scheduling in the hypervisor. We develop a profiling tool, vProfile, to decompose parallel runtime into three parts: compute, steal and synchronization. With the help of time breakdown, we devise two optimizations at the hypervisor to reduce slowdowns.

Combing the Communication Hairball: Visualizing Parallel Execution Traces using Logical Time.

With the continuous rise in complexity of modern supercomputers, optimizing the performance of large-scale parallel programs is becoming increasingly challenging. Simultaneously, the growth in scale magnifies the impact of even minor inefficiencies--potentially millions of compute hours and megawatts in power consumption can be wasted on avoidable mistakes or sub-optimal algorithms. This makes performance analysis and optimization critical elements in the software development process. One of the most common forms of performance analysis is to study execution traces, which record a history of per-process events and interprocess messages in a parallel application. Trace visualizations allow users to browse this event history and search for insights into the observed performance behavior. However, current visualizations are difficult to understand even for small process counts and do not scale gracefully beyond a few hundred processes. Organizing events in time leads to a virtually unintelligible conglomerate of interleaved events and moderately high process counts overtax even the largest display. As an alternative, we present a new trace visualization approach based on transforming the event history into logical time inferred directly from happened-before relationships. This emphasizes the code's structural behavior, which is much more familiar to the application developer. The original timing data, or other information, is then encoded through color, leading to a more intuitive visualization. Furthermore, we use the discrete nature of logical timelines to cluster processes according to their local behavior leading to a scalable visualization of even long traces on large process counts. We demonstrate our system using two case studies on large-scale parallel codes.