Behavior Of Parallel Programs Research Articles

Parallel programs execution optimization using behavior control in LuNA system

In the paper, the problem of efficient parallel execution of numerical algorithms for supercomputers in the LuNA system is concerned. With LuNA, an application algorithm is represented in a hardware-independent high-level form. This allows implementing the algorithm by automatic construction of various parallel programs, which possess different non-functional properties, such as execution time, memory consumption, network workload. In the LuNA system, the efficiency problem of automatically constructed parallel programs is dealt with through the behavior concept. The presented approach allows controlling parallel program behavior without low-level programming of the desired behavior.

Pedagogy and tools for teaching parallel computing at the sophomore undergraduate level

As the need for multicore-aware programmers rises in both science and industry, Computer Science departments in universities around the USA are having to rethink their parallel computing curriculum. At Rice University, this rethinking took the shape of COMP 322, an introductory parallel programming course that is required for all Bachelors students. COMP 322 teaches students to reason about the behavior of parallel programs, educating them in both the high level abstractions of task-parallel programming as well as the nitty gritty details of working with threads in Java.In this paper, we detail the structure, principles, and experiences of COMP 322, gained from 6 years of teaching parallel programming to second-year undergraduates. We describe in detail two particularly useful tools that have been integrated into the curriculum: the HJlibparallel programming library and the Habanero Autograder for parallel programs. We present this work with the hope that it will help augment improvements to parallel computing education at other universities.

Ordering Traces Logically to Identify Lateness in Message Passing Programs

Event traces are valuable for understanding the behavior of parallel programs. However, automatically analyzing a large parallel trace is difficult, especially without a specific objective. We aid this endeavor by extracting a trace's logical structure, an ordering of trace events derived from happened-before relationships, while taking into account developer intent. Using this structure, we can calculate an operation's delay relative to its peers on other processes. The logical structure also serves as a platform for comparing and clustering processes as well as highlighting communication patterns in a trace visualization. We present an algorithm for determining this idealized logical structure from traces of message passing programs, and we develop metrics to quantify delays and differences among processes. We implement our techniques in Ravel, a parallel trace visualization tool that displays both logical and physical timelines. Rather than showing the duration of each operation, we display where delays begin and end, and how they propagate. We apply our approach to the traces of several message passing applications, demonstrating the accuracy of our extracted structure and its utility in analyzing these codes.

PROBABILISTIC ANALYSIS OF LOAD-IMBALANCED PARALLEL APPLICATIONS WITH PARTIALLY ELIMINATED BARRIERS

In order to reduce the overhead of barrier synchronization, we have proposed an algorithm which eliminates barrier synchronizations and evaluated its validity experimentally in our previous study. As a result, we have found that the algorithm is more effective to the load-imbalanced program than load-balanced program. However, the degree of the load balance has not been discussed quantitatively. In this paper, we model the behavior of parallel programs. In our model, the execution time of a phase contained in a parallel program is represented as a random variable. To investigate how the degree of the load balance influences the performance of our algorithm, we varied the coefficient of variation (CV) of probability distribution which the random variable follows. Using the model, we evaluated the execution time of parallel programs which have four typical dependency patterns. Based on results, we found that theoretical results are consistent with experimental ones.

Data mining for defects in multicore applications: an entropy‐based call‐graph technique

SUMMARYMulticore computers are ubiquitous. Expert developers as well as developers with little experience in parallelism are now asked to create multithreaded software to exploit parallelism in mainstream shared‐memory hardware. However, finding and fixing parallel programming errors is a complex and arduous task. Programmers thus rely on tools such as race detectors that typically focus on reporting errors due to incorrect usage of synchronization constructs or due to missing synchronization. This arsenal of debugging techniques, however, is incomplete. This article presents a new perspective and addresses a largely unexplored direction of defect localization where a wrong usage of nonparallel programming constructs might cause wrong parallel application behavior. In particular, we make a contribution by showing how to use data‐mining techniques to locate defects in multithreaded shared‐memory programs. Our technique analyzes execution anomalies in a condensed representation of the dynamic call graphs of a multithreaded object‐oriented application and identifies methods that contain a defect. Compared with race detectors that concentrate on finding incorrect synchronization, our method is able to reveal a wider range of defects that affect the control flow of a parallel program. Results from controlled experiments show that our data‐mining approach finds not only race conditions in different types of multicore applications but also other errors that cause incorrect parallel program behavior. Data‐mining techniques offer a fruitful new ground for parallel program debugging, and we also discuss long‐term directions for this interesting field. Copyright © 2012 John Wiley & Sons, Ltd.

First-class monadic schedules

Parallel functional languages often use meta-linguistic annotations to provide control over parallel evaluation. In this paper we explore a flexible mechanism to control when an expression is evaluated: first-class monadic schedules . We discuss the advantages of using such first-class values over traditional annotation-based systems. In particular, it is often desirable to make decisions about the operational behavior of parallel programs depending on the dynamic state of the system. For example, we may want to measure the system load before deciding to evaluate expressions in parallel. For this purpose, we show how monads can be used to access dynamic system parameters in a referentially transparent manner (up to termination).As a mechanism to reason about schedules, we present a set of algebraic properties that any implementation of schedules must satisfy. We also describe an implementation that translates schedules into a dialect of Scheme extended with futures . We prove that this implementation satisfies the given set of algebraic properties, and give performance results for a parallel solution to the n -body problem using the Barnes--Hut method.Although our ideas were developed specifically for nonstrict functional languages such as Haskell, we briefly discuss how they can be used with strict functional languages and imperative languages as well.

Parallel elementwise processable functions in concurrent clean

The behaviour of concurrent and parallel programs can be specified in a functional style. Functional programming style has some inherent concurrent features. However, for a higher degree of expressing parallelism there is a need for new language constructs. In this paper, we introduce Concurrent Clean modules for evaluation strategies in order to control the evaluation degree, the dynamic behaviour, and the parallelism. The usage of the strategies will be illustrated by the parallel elementwise processing method. The implementation of the method in the lazy functional programming language Concurrent Clean operates with a two arguments two values elementwise processable function. In order to obtain abstract type specification for the generalised manipulation of the linear data structures like lists, arrays, and strict arrays, a linear data structure class module is defined. The programming style is skeleton based. Skeletons in functional languages are higher-order functions. The skeleton given in this paper represents a generalisation of the map function. It is triply parameterized: by an elementwise processable function, by type specification, and by the strategy parameter that defines the dynamic behaviour of the program.

PREFACE: Special Issue on High-Level Parallel Programming and Applications

Many applications of computing require performance levels attainable only on parallel architectures. Such systems are now readily available as their price/performance ratio continues to improve. Despite this, corresponding software has not developed to the same levels of capability, price, and flexibility as those of sequential software. There are deep technical reasons for this, which research has been addressing for two decades. Sequential programming has long benefited from high-level programming techniques and tools that have made today's immense range of software economically viable. Research into high-level parallel programming is producing methods and tools that improve the price/performance ratio of parallel software, and broaden the range of target applications. This special issue of Parallel Processing Letters presents recent work of researchers in this field. One promising approach to developing portable parallel programs lies in the use of algorithmic skeletons templates for common patterns of parallel computation. Jocelyn SEROT demonstrates the use of dataflow techniques for efficient implementation of nested dynamic skeletons. Andrea ZAVANELLA applies the BSP cost model to data-parallel skeletons, allowing architecture-independent optimizations. The BSP computation model is also the subject of contributions by Alexandre TISKIN, who shows how threads may be used to allow divide-and-conquer algorithms to be expressed in BSP. Frederic LOULERGUE, who presents a functional calculus of BSP programs ; and Yifeng CHEN & J. W. SANDERS who define a refinement calculus for deriving BSP programs from specifications. Paul H. J. KELLY, Olav BECKMANN, Tony FIELD & Scott B. BADEN present the THEMIS model for dynamic optimizations of run-time parallel program behavior. Hans-Wolfgang LOIDL, Philip W. TRINDER & Carsten BUTZ introduce a way of improving task granularity and data locality in parallel Haskell programs based on systematic transformations of a data class. Zineb H ABB AS, Michael KRAJECKI & Daniel SINGER show a simple decomposition strategy for distributing a constraint satisfaction problem among machines. These papers were first presented at the International Workshop on High-Level Parallel Programming and Applications in Orleans in March 2001. This special

Using spreadsheets as a simple and effective teaching tool for predicting and visualizing parallel program performance

This paper describes how we can use a spreadsheet program as an inexpensive and readily available tool to help students understand the execution behavior of parallel programs. Using the proposed method a performance prediction model can be created in a matter of minutes and the performance of the parallel system can be visualized and animated by using standard three-dimensional surface plots and user interface control objects.

The impact of I/O on program behavior and parallel scheduling

In this paper we systematically examine various performance issues involved in the coordinated allocation of processor and disk resources in large-scale parallel computer systems. Models are formulated to investigate the I/O and computation behavior of parallel programs and workloads, and to analyze parallel scheduling policies under such workloads. These models are parameterized by measurements of parallel programs, and they are solved via analytic methods and simulation. Our results provide important insights into the performance of parallel applications and resource management strategies when I/O demands are not negligible.

Interactive debugging and performance analysis of massively parallel applications

In the field of high performance computing, massively parallel processing systems (MPPs) get more and more important. A rising number of complex applications is parallelized for execution on these machines. Still a significant portion of the time needed for parallelization is spent for the process of debugging and performance tuning. A main reason for this fact is the absence of adequate tools supporting this phase of program development. In this article, we present a novel tool environment, consisting of a parallel debugger (DETOP), a performance analyzer (PATOP), and a common monitoring system for PowerPC-based parallel computers. The environment specifically addresses the topics of scalability, usability for dynamic, multithreaded programming models, minimal intrusion, debugging and tuning methodology and comfortable user interfaces. We derive requirements for tools monitoring the runtime behavior of parallel programs, present the concepts used to meet these requirements in our tool environment, and describe its implementation and its usage. DETOP is based on the event-action paradigm and supports both data parallel codes and programs based on functional decomposition. Special features are provided for applications that dynamically create new threads or consist of multiple executables. PATOP supports a systematic search for performance bottlenecks in massively parallel applications using the concept of attributed measurements and distributed evaluation. Both tools are based on a common, distributed on-line monitoring system providing the necessary runtime information.

The dynamic behaviour of parallel programs under process migration

AbstractWe have studied the interaction between process‐based parallel programs whose characteristics change in various ways at run time and the operation of load‐balancing, as implemented by process migration. In order to do this, we propose a simple performance model, whose parameters represent features of the program's execution such as the frequency and regularity of the changes in computational characteristics, and conduct a series of experiments involving simulated executions of synthetic programs with controlled parameter values. From these we can deduce the relative importance of the parameters from the point of view of their influence on performance.We can explain our observations in terms of a simplified stochastic model that relates local changes in load to global behaviour. We show that the dynamics of load‐balancing can be represented approximately by a first‐order difference equation, and that the distributed process migration algorithm is consistent with a behaviour on the global scale which can be regarded as that of a traditional feedback controller.

Performance analysis of parallel programs based on model calculations

Modeling parallel programs can help the developer during the design phase of a particular implementation on the one hand and on the other hand provide principle insights which are needed to establish design principles for the development of parallel programs. It is important that the model used provides a sufficient but not too detailed representation of the parallel program. Precedence graphs provide an easy to understand representation of the structure of parallel programs. Information about the behavior of parallel programs during execution can be obtained by assigning distributions of the execution times of the subtasks to the nodes of these graphs. In this paper we introduce truncated θ- exponential polynomials as a class of distributions suitable for the modeling of task run-times of massively parallel programs. As opposed to classical exponential polynomials known from literature, truncated θ-exponential polynomials allow the representation of distributions with finite domain and are therefore appropriate for the analysis of massively parallel systems. We present approximation formulas which allow the approximate calculation of the total execution time for large models in particular. We provide formulas for the parallel execution of a large number of subtasks. We also give a method to estimate the total execution time of regular graphs, which are also very important in the field of modeling parallel programs.

Adaptive memory coherence algorithms in DSVM

Based on the characteristics of distrubuted system and the behavior of parallel programs, this paper presents the fixed and randomized competitive memory coherence algorithms for distributed shared virtual memory. These algorithms exploit parallel programs' locality of reference and exhibit good competitive property. Our simulation shows that the fixed and randomized algorithms achieve better performance and higher stability than other strategies such as write-invalidate and write-update.

Modelling parallel program behaviour in ALPES

The goal of this paper is to present ANDES, a language for modelling quantitative behaviour of parallel programs. ANDES is employed in the ALPES environment. From an ANDES quantitative model, a parallel synthetic program is generated. This synthetic program is executed on a real distributed memory parallel computer. Some performance indices are obtained from this execution. Currently, mapping strategies are being evaluated using this approach. The ALPES environment is a research domain in the APACHE project.

Scheduling of parallel programs including dynamic loops

The classical static scheduling strategies do not consider the dynamic behavior of parallel programs. These strategies provide near-optimal solutions for the scheduling problem when the programs are well-known beforehand. However, the schedulings provided can be quite inefficient when the conditions of a real execution are not those assumed to obtain the scheduling. On the other hand, some of the structures with a dynamic behavior included in parallel programs include an intrinsic parallelism (e.g., Do-All loops), that cannot be exploited by the classical scheduling policies. To avoid these situations, the authors developed a methodology based on Task Replication that keeps the processors as busy as possible and takes advantage of the intrinsic parallelism of parallel programs. This methodology provides static schedulings, with some replicated tasks, that take into account the dynamic behavior of parallel programs including loops with a variable number of iterations and conditional structures.

Analysis of the impact of memory in distributed parallel processing systems

We consider an important tradeoff between processor and memory allocation in distributed parallel processing systems. To study this tradeoff, we formulate stochastic models of parallel program behavior, distributed parallel processing environments and memory overheads incurred by parallel programs as a function of their processor allocation. A mathematical analysis of the models is developed, which includes the effects of contention for shared resources caused by paging activity. We conduct a detailed analysis of real large-scale scientific applications and use these results to parameterize our models. Our results show that memory overhead resulting from processor allocation decisions can have a significant effect on system performance in distributed parallel environments, strongly suggesting that memory considerations must be incorporated in the resource allocation policies for parallel systems. We also demonstrate the importance of the inter-locality miss ratio, which is introduced in this paper and analyzed for the first time.

Graphical Views of the Behavior of Parallel Programs

There is often an increasing gap between a programmer′s mental conception of a problem and the realization of a parallel program which solves that problem. To bridge this gap, information about program execution must be displayed. Since a parallel program may be executed on several processors simultaneously, activities may be distributed across both space and time, and therefore simple sequential displays of a program′s execution are inadequate. Intuitively, pictures are much more meaningful than text, particularly with parallel programs where large quantities of information may be generated. In this paper, we describe how the behavior of parallel programs may be graphically interpreted, and discuss current approaches and difficulties in parallel program visualization. A graphical interface designed for languages based on message-passing is also presented.

Visualization of Message Passing Parallel Programs with the TOPSYS Parallel Programming Environment

Parallel programming is orders of magnitudes more complex than writing sequential programs. This is particularly true for programming distributed memory multiprocessor architectures based on message passing programming models. Apart from understanding the sequential parts of the parallel program, new degrees of freedom lead to additional problems. Understanding the synchronization and communication behavior of parallel programs is the most critical issue in programming distributed memory multiprocessors. The paper describes methods and tools for visualization and animation of the dynamic execution of parallel programs. Based on an evaluation and classification of existing visualization environments, the visualization and animation tool VISTOP (VISualization TOol for Parallel Systems) is presented as part of the integrated tool environment TOPSY S (TOols for Parallel SYStems) for programming distributed memory multiprocessors. VISTOP supports the interactive on-line visualization of message passing programs based on various views; in particular, a process graph based concurrency view for detecting synchronization and communication bugs.

Breaking the Silence: Auralization of Parallel Program Behavior

The run-time behavior of a parallel program is defined by many parameters, for example, the program′s communication structure, the processor utilization profile, and the dynamic size of message queues. To understand the execution of a program, it is frequently necessary for a programmer to consider the run-time behavior from a number of different angles. Also, it is sometimes useful to consider one aspect of the behavior in isolation, whereas other times it is necessary to consider different types of behavior together. Sound offers an alternative form of investigation to simply using multiple graphical and textual views for studying the behavior of a program. In this paper, we discuss the properties of parallel programs that are well suited to being mapped to sound and present a number of example mappings. As evidence of the effectiveness of the sound mappings, we present case studies based on a prototype sound tool. In general, sound was found to be effective in depicting certain patterns and timing information related to the programs′ behaviors. Also, by listening to sound representations that were sychronized with graphical displays, the speed of recognition and distinction of programs, and parts of programs, was increased.