Execution Time Of Parallel Program Research Articles

The Energy Efficiency Evaluating Method Determining Energy Consumption of the Parallel Program According to Its Profile

The paper is devoted to the evaluation of energy efficiency of High Performance Computing systems used in a scientific supercomputer center. The authors propose a method for the comparison of energy efficiency of computing systems based on the power consumption and execution time of parallel programs. The paper presents a software tool that allows to determine the energy consumption profile of a parallel program automatically without changing its source code. The paper also presents the results of power consumption comparison of NAS Parallel Benchmarks (BT, EP, IS, and LU) tests on computing systems with codenames Intel microprocessors Broadwell, Cascade Lake, Knights Landing and Skylake).

Multilevel procedure for decomposition and mapping graphs

The paper analyzes the actual problem of architecture-dependent decomposition, allowing efficient planning of running a parallel task in a multiprocessor computer network. The procedure of planning means decomposition of a parallel task in a required number of processors accounting for balance restrictions and mapping of obtained portions to processors in order to minimize interprocessor communication costs. This procedure allows to reduce parallel program execution time and consequently expensive hardware usage time. The paper provides general mathematical formulation of this problem, and special cases are considered. Multilevel algorithms are suggested for a special and a general case of the problem. Results of experiments in well-known benchmarks as compared to renowned software METIS and SCOTCH.

Automated Performance Modeling of HPC Applications Using Machine Learning

Automated performance modeling and performance prediction of parallel programs are highly valuable in many use cases, such as in guiding task management and job scheduling, offering insights of application behaviors, and assisting resource requirement estimation. The performance of parallel programs is affected by numerous factors, including but not limited to hardware, applications, algorithms, and input parameters, thus an accurate performance prediction is often a challenging and daunting task. In this article, we focus on automatically predicting the execution time of parallel programs (more specifically, MPI programs) with different inputs, at different scales, and without domain knowledge. We model the correlation between the execution time and domain-independent runtime features. These features include values of variables, counters of branches, loops, and MPI communications. Through automatically instrumenting an MPI program, each execution of the program will output a feature vector and its corresponding execution time. After collecting data from executions with different inputs, a random forest machine learning approach is used to build an empirical performance model, which can predict the execution time of the program given a new input. A transfer learning method is used to reuse an existing performance model and improve the prediction accuracy on a new platform that lacks historical execution data. Our experiments and analyses of three parallel applications, Graph500, GalaxSee, and SMG2000, on three different systems confirm that our method performs well, with less than 20 percent prediction error on average.

Predicting HPC parallel program performance based on LLVM compiler

Performance prediction of parallel program plays key roles in many areas, such as parallel system design, parallel program optimization, and parallel system procurement. Accurate and efficient performance prediction on large-scale parallel systems is a challenging problem. To solve this problem, we present an effective framework for performance prediction based on the LLVM compiler technique in this paper. We can predict the performance of a parallel program on a small amount of nodes of the target parallel system using this framework toned but not execute this parallel program on a corresponding full-scale parallel system. This framework predicts the performance of computation and communication components separately and combines the two predictions to achieve full program prediction. As for sequential computation, we first combine the static branch probability and loop trip count identification and propose a new instrumentation method to acquire the number of each instruction type. We then construct a test program to measure the average execution time of each instruction type. Finally, we utilize the pruning technique to convert a parallel program into a corresponding sequential program to predict the performance on only one node of the target parallel system. As for communication, we utilize the LogGP model to model point-to-point communication and the artificial neural network technique to model collective communication. We validate our approach by a set of experiments that predict the performance of NAS parallel benchmarks and CGPOP parallel application. Experimental results show that the proposed framework can accurately predict the execution time of parallel programs, and the average error rate of these programs is 10.86%.

PROBABILISTIC ANALYSIS OF LOAD-IMBALANCED PARALLEL APPLICATIONS WITH PARTIALLY ELIMINATED BARRIERS

In order to reduce the overhead of barrier synchronization, we have proposed an algorithm which eliminates barrier synchronizations and evaluated its validity experimentally in our previous study. As a result, we have found that the algorithm is more effective to the load-imbalanced program than load-balanced program. However, the degree of the load balance has not been discussed quantitatively. In this paper, we model the behavior of parallel programs. In our model, the execution time of a phase contained in a parallel program is represented as a random variable. To investigate how the degree of the load balance influences the performance of our algorithm, we varied the coefficient of variation (CV) of probability distribution which the random variable follows. Using the model, we evaluated the execution time of parallel programs which have four typical dependency patterns. Based on results, we found that theoretical results are consistent with experimental ones.

클러스터 상에서 다중 코어 인지 부하 균등화를 위한 Chapel 데이터 분산 구현

클러스터와 같은 분산 메모리 구조에서 각 노드는 전체 데이터의 일부분을 저장하고 있다. 이러한 구조에서는 데이터를 각 노드에 분산시키는 방법이 성능에 영향을 준다. 데이터 분산 정책은 데이터를 노드들에게 분산시켜 병렬 데이터 처리를 실현하는 정책이다. 클러스터 관리, 확장, 업그레이드 등 다양한 요인으로 인해 클러스터의 각 노드 성능이 동일하지 않을 수 있다. 이러한 클러스터에서 노드의 성능을 고려하지 않은 데이터 분산 정책은 데이터를 각 노드에 효율적으로 분산시키지 못할 수 있다. 본 논문에서는 각 노드의 성능을 나타내는 인자로 노드에 장착되어 있는 프로세서의 코어 수를 이용하고, 이를 고려한 데이터 분산 정책을 제안한다. 본 논문에서 제안하는 데이터 분산 정책에서는 전체 코어 수 대비 노드에 장착된 코어 수에 비례하여 데이터를 노드에 분산 저장하도록 할당을 한다. 또, 본 논문에서 제안하는 데이터 분산 정책을 Chapel 언어를 이용하여 구현하였다. 본 논문에서 제안하는 데이터 분산 정책이 효과적임을 입증하기 위해 이 정책을 이용하여 Mandelbrot 집합과 원주율을 계산하는 병렬 프로그램을 작성하고, 클러스터에서 실행하여 실행 시간을 비교한다. 8-코어와 16-코어로 구성되어 있는 클러스터에서 수행한 결과에 의하면 노드의 코어 수를 기반으로 한 데이터 분산 정책이 병렬 프로그램의 수행 시간 감소에 기여하였다. In distributed memory architectures like clusters, each node stores a portion of data. How data is distributed across nodes influences the performance of such systems. The data distribution scheme is the strategy to distribute data across nodes and realize parallel data processing. Due to various reasons such as maintenance, scale up, upgrade, etc., the performance of nodes in a cluster can often become non-identical. In such clusters, data distribution without considering performance cannot efficiently distribute data on nodes. In this paper, we propose a new data distribution scheme based on the number of cores in nodes. We use the number of cores as the performance factor. In our data distribution scheme, each node is allocated an amount of data proportional to the number of cores in it. We implement our data distribution scheme using the Chapel language. To show our data distribution is effective in reducing the execution time of parallel applications, we implement Mandelbrot Set and <TEX>${\pi}$</TEX>-Calculation programs with our data distribution scheme, and compare the execution times on a cluster. Based on experimental results on clusters of 8-core and 16-core nodes, we demonstrate that data distribution based on the number of cores can contribute to a reduction in the execution times of parallel programs on clusters.

Расширение модели ParJava для случая кластеров с многоядерными узлами

At the beginning of 2000 each node of high performance cluster with distributed memory contained processor with single core and each MPI process of parallel application used to utilize all resources of node. At this moment vendors propose microprocessors with multiple cores on chip and few processors on the single board. Using multiple threads in a single node with modern multicore processors allows to increase performance of parallel application due to usage of shared memory and lower overhead. An extension of model for parallel SPMD programs has been developed with ability to use Java threads. The usage of threads in program allows better utilization of the resources of multicore processor. Developed model allows estimate execution time of parallel program with explicit calls to MPI library, where parallel Java threads could be used in each process. However, there are a lot of problems arising when threads have been used in Java environment. This paper contains recommendations called for performance tuning of multiprocessed-multithreaded program concerning to JVM memory management, garbage collector configuration, management of local buffers etc. Java version of parallel application FT (Fast Furier Transformation) from NPB has been adapted for multiprocess-multithreaded environment. Tests on implemented application show 9-14% performance improvement. Model of multiprocess-multithreaded application has been developed. Performance prediction for multiprocess-multithreaded FT shows 3-7% prediction error.

Parallel execution time prediction of the multitask parallel programs

A critical problem of predicting the execution time of parallel programs is computing the maximum execution time of tasks involved in the parallel computation. For a parallel composition of n tasks, distribution information of execution time of the constituent task is crucial to accurately predicting mean, variance and even distribution of execution time of the composition of tasks when the execution time of them is stochastic. This paper presents a method of predicting the maximum parallel execution time of n constituent tasks, each of which has independent, identically distributed random execution time. First, execution time of any constituent task as a random variable is transformed into standard normal variable, or approximately so, by using Johnson distributions. Then the distribution and moments of the maximum parallel execution time are obtained after appropriate Johnson transformation is chosen and its corresponding parameters are determined. The experiment on synthetic distributions such as exponential distribution shows that most relative errors of the first four moments are near the 0.1%. And the experiment on some practical applications shows that the relative errors of the first four moments are below 0.1%. The computing complexity of our algorithm, as a function of the number n of the tasks or processors, is O ( n ) , and even O ( 1 ) while using approximate distribution, such as Gumbel distribution.

Predicting the execution times of parallel-independent programs using Pearson distributions

Predicting the execution time of parallel programs involves computing the maximum or minimum of the execution times of the tasks involved in the parallel computation. We present a method to accurately compute the distribution of the largest (Max) and the smallest (Min) execution time of the composite of a number of parallel programming tasks, each having an independent, stochastic, arbitrary workload. The Max function applies to the general case that the composite task completes at the time its longest constituent task terminates. The Min function applies when the completion of its shortest task terminates the whole parallel process, such as in a parallel searching program. Both the Min and Max density function of a constituent task are characterized in terms of a Pearson distribution. Due to its accuracy, the presented method is especially of interest when the performance of time critical parallel applications must be derived. Both prediction methods are tested against three well-known distributions. Furthermore, the Max prediction method is also tested against a number of measured real-life data parallel programs with different degree of parallelism. The results show excellent accuracy of better than 1% with a very few exceptions in extreme situations.

이기종 클러스터를 위한 수정된 GSS 부하 분할 알고리즘

클러스터는 컴퓨터 네트워크로 연결되어 있는 컴퓨터들로 구성된 비용대비 효과적인 병렬 처리 환경이다. 클러스터의 특징으로는 노드의 이기종성, 부하의 다양성, 네트워크 부하의 다양성 등이다. 이러한 특징들은 병렬 프로그램의 수행 성능에 영향을 주기 때문에 클러스터를 위한 부하 분할은 병렬 프로그램의 성능에 많은 영향을 준다. 본 논문에서는 부하 분할 알고리즘인 GSS를 수정한 αGSS 알고리즘을 제안한다. 본 논문에서 제안하는 αGSS 알고리즘에서는 각 노드가 처리할 태스크의 크기를 결정할 때 각 노드의 BogoMIPS를 이용한다. 실험 결과에 의하면 제안된 αGSS 알고리즘이 이기종으로 구성된 클러스터에서 효과적으로 부하를 분할하며, 따라서 병렬 프로그램의 수행 시간을 감소시킬 수 있다. Cluster is the cost-effective parallel processing environment, and consists of the off-the-shelf computers connected by the computer net works. The characteristics of cluster are the node heterogeneity, the variety of node load, and the variety of network load. Because these characteristics influence the performance of parallel program executions, the load sharing for cluster is important, and by using the proper load sharing strategy, we can reduce the execution time of parallel programs. In this paper, we propose modified GSS algorithm, αGSS. In the proposed load sharing algorithms α GSS, the size of tasks are decided using the BogoMIPS of node. From the result of out experiments, we conclude that the proposed αGSS algorithm is effective in the heterogeneous cluster.

A Simple Method for Dynamic Scheduling in a Heterogeneous Computing System

A simple method for the dynamic scheduling on a heterogeneous computing system is proposed in this paper. It was implemented to minimize the parallel program execution time. The proposed method decomposes the program workload into computationally homogeneous subtasks, which may be of the different size, depending on the current load of each machine in a heterogeneous computing system.

Using parallel program characteristics in dynamic processor allocation policies

Abstract In multiprocessors a parallel program's execution time is directly influenced by the number of processors it is allocated. The problem of scheduling parallel programs in a multiprogrammed environment becomes one of determining how to best allocate processors to the different simultaneously executing programs in order to minimize mean response time. In this paper we address the problem of how many processors to allocate to each of the executing parallel jobs by examining the following questions: 1. (1) Is allocating processors equally among all jobs (equipartitioning) a desirable property of a scheduling algorithm? 2. (2) Does using information about the service demand of parallel jobs significantly reduce mean response time? 3. (3) Does using information about the efficiency with which parallel jobs execute significantly reduce mean response time? 4. (4) Does allocating each job a number of processors corresponding to the knee of the execution time — efficiency curve significantly reduce mean response time? 5. (5) What are the desirable properties of a scheduler that is designed to minimize mean response time? The main contributions of this paper are: a first-order understanding of how processing power should be dynamically allocated to jobs, a new family of algorithms for dynamic processor allocation and a rough quantification of the benefits that may be realized by properly utilizing job characteristics when making processor allocation decisions. We believe that these new algorithms can be combined with recently demonstrated techniques for obtaining sufficiently accurate runtime estimates of job efficiencies to improve scheduler implementations for multiprogrammed multiprocessors.

Using parallel program characteristics in dynamic processor allocation policies

In multiprocessors a parallel program's execution time is directly influenced by the number of processors it is allocated. The problem of scheduling parallel programs in a multiprogrammed environment becomes one of determining how to best allocate processors to the different simultaneously executing programs in order to minimize mean response time. In this paper we address the problem of how many processors to allocate to each of the executing parallel jobs by examining the following questions: 1. (1) Is allocating processors equally among all jobs (equipartitioning) a desirable property of a scheduling algorithm? 2. (2) Does using information about the service demand of parallel jobs significantly reduce mean response time? 3. (3) Does using information about the efficiency with which parallel jobs execute significantly reduce mean response time? 4. (4) Does allocating each job a number of processors corresponding to the knee of the execution time — efficiency curve significantly reduce mean response time? 5. (5) What are the desirable properties of a scheduler that is designed to minimize mean response time? The main contributions of this paper are: a first-order understanding of how processing power should be dynamically allocated to jobs, a new family of algorithms for dynamic processor allocation and a rough quantification of the benefits that may be realized by properly utilizing job characteristics when making processor allocation decisions. We believe that these new algorithms can be combined with recently demonstrated techniques for obtaining sufficiently accurate runtime estimates of job efficiencies to improve scheduler implementations for multiprogrammed multiprocessors.

Accurate Predictions of Parallel Program Execution Time

It has been difficult to develop simple formulations to predict the execution time of parallel programs due to the complexity of characterizing parallel hardware and software. In an attempt to clarify these characterizations, we introduce a methodology for applying a simple performance model based on Amdahl′s law. Our formulation results in accurate predictions of execution time on available systems, allowing programmers to select the optimal number of processors to apply to a particular problem or to select an appropriate problem size for the number of processors available. In short, we accurately quantify the scalability of a specific algorithm when it is run on a specific parallel computer. Our predictions are based on simple experiments that characterize machine performance and on a simple analysis of the parallel program. We illustrate our method for a program executed on a Sequent Symmetry multiprocessor with 20 processors. Our predictions closely match experimental results, differing by no more than 5% from the actual execution times. Our results illustrate key performance limitations of parallel systems, showing the impact of overhead and the scaling of problem size.

Stochastic performance models of parallel task systems (extended abstract)

This paper considers the class of parallel computations represented by directed, acyclic task graphs. These include parallel loops, multiphase algorithms, partitioning and merging algorithms, as well as any arbitrary parallel computation that can be structured by a task graph. The paper reviews the current state of the art in stochastic bound models of parallel programs and presents new stochastic bound performance models that predict the expected execution time of parallel programs on a given shared-memory multiprocessor system; and provide qualitative and quantitative description of the relationships between the structure of parallel programs, computation and synchronization behavior of the program, and architectural features of the underlying multiprocessor system. The models use a new formulation based on stochastic bound analysis and are solvable for a number of distribution functions. They are applicable to shared-memory multiprocessors with significantly different architectural and synchronization performance characteristics. The accuracy of the models is validated via several measurements on two different shared-memory multiprocessor systems, the Alliant FX/2800 and the Encore Multimax. The results show the models to be quite accurate, even when some of the modeling assumptions are violated. The maximum error of prediction ranges from about 10% to under 1%.

Timing parallel programs that use message passing

Hoare's paper describing the theoretical framework in which to express algorithms for parallel programs using message passing has led to the development ofmany parallel processing systems. As more and more parallel algorithms are developed and implemented, there is a rising need for a mechanism to compare their relative speedup and scalability. Conventional techniques for accumulating timing data for sequential programs are not powerful enought to solve the special problems introduced by parallelism. In this paper we present a timing metholodolgy capable of accurately determining the execution time of parallel programs, independent of system loads. Further, this timing methodology is capable of acquiring timing data when the number of processes exceeds the number of physical CPUs. Through simulation this methodology also provides a means by which to study the behavior of a parallel program under an arbitrary hardware/software communications configuration.

Compile-time partitioning of iterative parallel loops to reduce cache coherency traffic

Adaptive data partitioning (ADP) which reduces the execution time of parallel programs by reducing interprocessor communication for iterative parallel loops is discussed. It is shown that ADP can be integrated into a communication-reducing back end for existing parallelizing compilers or as part of a machine-specific partitioner for parallel programs. A multiprocessor model to analyze program execution factors that lead to interprocessor communication and a model for the iterative parallel loop to quantify communication patterns within a program are defined. A vector notation is chosen to quantify communication across a global data set. Communication parameters are computed by examining the indexes of array accesses and are adjusted to reflect the underlying system architecture by compensating for cache line sizes. These values are used to generate rectangular and hexagonal partitions that reduce interprocessor communication.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Stochastic bounds on execution times of parallel programs

Stochastic bounds are obtained on execution times of parallel programs when the number of processors is unlimited. A parallel program is considered to consist of interdependent tasks with synchronization constraints. These constraints are described by an acyclic directed graph called a task graph. The execution times of tasks are considered to be independently identically distributed (i.i.d.) random variables. The performance measure of interest is the overall execution of the considered parallel program (task graph). Stochastic bound methods are applied to obtain lower and upper bounds on this measure. Another upper bound is obtained for parallel programs having 'new better than used in expectation' (NBUE) random variables as task execution times. NBUE random variables are replaced with exponential random variables of the same mean to derive this upper bound. >

Compiler techniques for data partitioning of sequentially iterated parallel loops

This paper uses bottom-up, static program partitioning to minimize the execution time of parallel programs by reducing interprocessor communication. Program partitioning is applied to a parallel programming construct known as a sequentially iterated parallel loop. This paper develops and evaluates compiler techniques to automatically generate data partitions for sequentially iterated parallel loops that minimize interprocessor communication. These techniques could be included as a communication-reducing back end for existing parallelizing compilers. A multiprocessor model is defined to analyze program execution factors that lead to interprocessor communication. A model for the sequentially iterated parallel loop is defined to quantify communication patterns within a program. A vector notation is chosen to quantify communication across a global data set. Communication parameters used in generating data partitions are found and used within equations which are shown to generate rectangular and hexagonal partition shapes that minimize interprocessor communication. Experiments measuring the impact of communication on program performance and the improvements gained by using the automatically generated data partitions were conducted on a sixteen-processor Encore Multimax. The new method outperformed established data partitioning methods.