Scalable Parallel Systems Research Articles

MPI performance engineering with the MPI tool interface: The integration of MVAPICH and TAU

Abstract The desire for high performance on scalable parallel systems is increasing the complexity and tunability of MPI implementations. The MPI Tools Information Interface (MPI_T) introduced as part of the MPI 3.0 standard provides an opportunity for performance tools and external software to introspect and understand MPI runtime behavior at a deeper level to detect scalability issues. The interface also provides a mechanism to fine-tune the performance of the MPI library dynamically at runtime. In this paper, we propose an infrastructure that extends existing components — TAU, MVAPICH2, and BEACON to take advantage of the MPI_T interface and offer runtime introspection, online monitoring, recommendation generation, and autotuning capabilities. We validate our design by developing optimizations for a combination of production and synthetic applications. Using our infrastructure, we implement an autotuning policy for AmberMD (a molecular dynamics package) that monitors and reduces the internal memory footprint of the MVAPICH2 MPI library without affecting performance. For applications such as MiniAMR whose collective communication is latency sensitive, our infrastructure is able to generate recommendations to enable hardware offloading of collectives supported by MVAPICH2. By implementing this recommendation, the MPI time for MiniAMR at 224 processes reduces by 15%.

Performance Analysis and Optimization of a Hybrid Seismic Imaging Application

Applications to process seismic data are computationally expensive and, therefore, employ scalable parallel systems to produce timely results. Here we describe our experiences of using performance analysis tools to gain insight into an MPI+OpenMP code developed by Shell that performs Reverse Time Migration on a cluster to produce models of the subsurface. Tuning MPI+OpenMP programs for modern platforms is difficult, and, therefore, assistance is required from performance analysis tools. These tools provided us with insights into the effectiveness of the domain decomposition strategy, the use of threaded parallelism, and functional unit utilization in individual cores. By applying insights obtained from Rice University's HPCToolkit and hardware performance counters, we were able to improve the performance of Shell's prototype distributed-memory Reverse Time Migration code by roughly 30 percent.

Retargeting of the Open Community Runtime to Intel Xeon Phi

Abstract The Open Community Runtime (OCR) is a recent effort in the search for a runtime for extreme scale parallel systems. OCR relies on the concept of a dynamically generated task graph to ex- press the parallelism of a program. Rather than being directly used for application development, the main purpose of OCR is to become a low-level runtime for higher-level programming models and tools. Since manycore architectures like the Intel Xeon Phi are likely to play a major role in future high performance systems, we have implemented the OCR API for shared-memory machines, including the Xeon Phi. We have also implemented two benchmark applications and performed experiments to investigate the viability of the OCR as a runtime for manycores. Our initial experiments and a comparison with OpenMP indicate that OCR can be an efficient runtime system for current and emerging manycore systems.

A Practical Data Classification Framework for Scalable and High Performance Chip-Multiprocessors

State-of-the-art chip multiprocessor (CMP) proposals emphasize general optimizations designed to deliver computing power for many types of applications. Potentially, significant performance improvements that leverage application-specific characteristics such as data access behavior are missed by this approach. In this paper, we demonstrate how scalable and high-performance parallel systems can be built by classifying data accesses into different categories and treating them differently. We develop a novel compiler-based approach to speculatively detect a data classification termed practically private, which we demonstrate is ubiquitous in a wide range of parallel applications. Leveraging this classification provides efficient solutions to mitigate data access latency and coherence overhead in today’s many-core architectures. While the proposed data classification scheme can be applied to many micro-architectural constructs including the TLB, coherence directory, and interconnect, we demonstrate its potential through an efficient cache coherence design. Specifically, we show that the compiler-assisted mechanism reduces an average of 46% coherence traffic and achieves up to 12%, 8%, and 5% performance improvement over shared, private, and state-of-the-art NUCA-based caching, respectively, depending on scenarios.

Editorial: enabling technologies for programming extreme scale systems

Multi-core architecture presents a new trend, and core-based parallel processing algorithms will continue to become more important. In addition, Grid and Cloud Computing (GCC) has emerged rapidly as an exciting new paradigm that offers a challenging model of computing and poses fascinating problems ranging from parallel architecture to programming models and compiler optimization. Furthermore, the wide-scale deployment of computing nodes, clusters, and embedded processors will continue to fuel core parallel processing and distributed systems algorithms. Thus, the importance of scalable parallel programming will continue to grow. The field of parallel and distributed computing is going through rapid changes. The emerging multi-core commodity processors and advanced networking technologies are allowing for the design of cost-effective parallel and distributed systems. To achieve scalability and high performance in these systems, it is increasingly becoming challenging to design better architecture, software environments, networking protocols, compilers, operating systems, languages, algorithms and applications for designing next generation scalable and high-performance parallel systems. This special issue is intended to foster stateof-the-art research in the area of extreme scale parallel systems including the topics of parallel algorithm design, data flow analysis, virtualization, energy efficient, load balancing, parallel computing model, multi-core programming, GPU implementation and optimization, execution on real-world parallel architecture and novel applications associated with this new paradigm. The papers included in this special issue demonstrate the effectiveness and efficiency of a variety of technologies and applications in parallel, grid, cloud and extreme scale systems. All of them not only contribute valuable insights and results but

On A New Interconnection Network for Large Scale Parallel Systems

This paper proposes a new cube based topology called the Folded Metacube (FMC). The new topology has attractive features such as reduced diameter, cost and improved broadcast time in comparison to the Metacube. Two separate routing algorithms one-to-one and one-to-all broadcast are proposed for the new network. Performance analysis such as cost effectiveness, time-cost-effectiveness and reliability of the new topology are carried out. The proposed network exhibits noticeable improvement in terms of the topological parameters and is superior to the other existing cube based networks. General Terms Parallel Computing, Performance analysis

Scalability and Communication in Parallel Low-Complexity Lossless Compression

Approximation schemes for optimal compression with static and sliding dictionaries which can run on a simple array of processors with distributed memory and no interconnections are presented. These approximation algorithms can be implemented on both small and large scale parallel systems. The sliding dictionary method requires large size files on large scale systems. As far as lossless image compression is concerned, arithmetic encoders enable the best lossless compressors but they are often ruled out because they are too complex. Storer extended dictionary text compression to bi-level images to avoid arithmetic encoders (BLOCK MATCHING). We were able to partition an image into up to a hundred areas and to apply the BLOCK MATCHING heuristic independently to each area with no loss of compression effectiveness. Therefore, the approach is suitable for a small scale parallel system at no communication cost. On the other hand, bi-level image compression seems to require communication on large scale systems. With regard to grey scale and color images, parallelizable lossless image compression (PALIC) is a highly parallelizable and scalable lossless compressor since it is applied independently to blocks of 8 × 8 pixels. We experimented the BLOCK MATCHING and PALIC heuristics with up to 32 processors of a 256 Intel Xeon 3.06 GHz processors machine (http://avogadro.cilea.it) on a test set of large topographic bi-level images and color images in RGB format. We obtained the expected speed-up of the compression and decompression times, achieving parallel running times about 25 times faster than the sequential ones. Finally, scalable algorithms computing static and sliding dictionary optimal text compression on an exclusive read, exclusive write shared memory parallel machine are presented. On the same model, compression by block matching of bi-level images is shown which can be implemented on a full binary tree architecture under some realistic assumptions with no scalability issues.

An interference-aware multichannel media access control protocol for wireless sensor networks

Scalable Parallel Systems (SPS) have offered a challenging model of computing and poses fascinating optimizations in sensor networks. With the development of sensor hardware technology, a certain sensor node is equipped with a radio transceiver that can be tuned to work on multiple channels. In this paper, we develop a novel interference-aware multichannel media access control (IMMAC) protocol for wireless sensor networks, which takes advantage of multichannel availability. Firstly, each node is assigned with a quiescent channel to reduce hidden terminal beforehand, and then it makes channel adjustment according to dynamic traffic. Secondly, a scalable multichannel media access control protocol is designed to make a tradeoff between channel switching overhead and fairness, and it effectively supports for node unicast and broadcast based on the receiver-directed channel switching. We have implemented simulation to evaluate the performance of IMMAC by comparing with other relevant protocols. The results show that our protocol exhibits more prominent ability, which utilizes multichannel to make parallel transmission and reduce hidden terminal problems effectively in resource-constrained wireless sensor networks.

A Pragmatic Analysis Of Scheduling Environments On New Computing Platforms

Today, large scale parallel systems are available at relatively low cost. Many powerful such systems have been installed all over the world and the number of users is always increasing. The use of such clusters requires special administration tools, which have been developed recently and most frequently rely on intuitive heuristics to solve the underlying difficult scheduling problems. On the other hand, recent theoretical work has designed a number of models, such as the Parallel Task model, specifically aimed at cluster environments. The objective of this work is to study and analyze the path from theoretical models and algorithms to their implementation on an actual environment on a real platform. We outline in detail the divergences between classical models and a real batch scheduling system, and propose solutions to adapt a theoretically well-founded algorithm to such a system. Experimental results show that this implementation performs about as well as FCFS with backfilling, and much better in the difficult instances. We hope that further usage of this algorithm in a live system will show that interaction between theory and practice can be fruitful.

SCHEDULING ON LARGE SCALE DISTRIBUTED PLATFORMS: FROM MODELS TO IMPLEMENTATIONS

Today, large scale parallel systems are available at low cost, Many powerful such systems have been installed all over the world and the number of users is always increasing. The difficulty of using them efficiently is growing with the complexity of the interactions between more and more architectural constraints and the diversity of the applications. The design of efficient parallel algorithms has to be reconsidered under the influence of new parameters of such platforms (namely, cluster, grid and global computing) which are characterized by a larger number of heterogeneous processors, often organized in several hierarchical sub-systems. At each step of the evolution of the parallel processing field, researchers designed adequate computational models whose objective was to abstract the real world in order to be able to analyze the behavior of algorithms. In this paper, we will investigate two complementary computational models that have been proposed recently: Parallel Task (PT) and Divisible Load (DL). The Parallel Task (i.e. tasks that require more than one processor for their execution) model is a promising alternative for scheduling parallel applications, especially in the case of slow communication media. The basic idea is to consider the application at a coarse level of granularity. Another way of looking at the problem (which is somehow a dual view) is the Divisible Load model where an application is considered as a collection of a large number of elementary – sequential – computing units that will be distributed among the available resources. Unlike the PT model, the DL model corresponds to a fine level of granularity. We will focus on the PT model, and discuss how to mix it with simple Divisible Load scheduling. As the main difficulty for distributing the load among the processors (usually known as the scheduling problem) in actual systems comes from handling efficiently the communications, these two models of the problem allow us to consider them implicitly or to mask them, thus leading to more tractable problems. We will show that in spite of the enormous complexity of the general scheduling problem on new platforms, it is still useful to study theoretical models. We will focus on the links between models and actual implementations on a regional grid with more than 500 processors.

An analytical model of wormhole-routed hypercubes under broadcast traffic

A large number of algorithms have been proposed to support collective communication operations for scalable parallel systems over the past few years. When proposing a new algorithm for a collective communication operation, it is critical to determine its precise scope and evaluate it with accurate modelling of the underlying routing and communication mechanism. However, there has been comparatively little activity in the area of analytical models of these operations. As a result, most existing studies have relied on simulation to evaluate the performance merits of collective communication algorithms. This paper presents a new analytical model for predicting unicast and broadcast latency in the wormhole-routed hypercube. Results obtained through simulation experiments show that the model exhibits a good degree of accuracy in predicting message latency under different working conditions.

Parallel Strategies for Rank-k Updating of the QR Decomposition

Parallel strategies based on Givens rotations are proposed for updating the QR decomposition of an n × n matrix after a rank-k change (k < n). The complexity analyses of the Givens algorithms are based on the total number of Givens rotations applied to a 2-element vector. The algorithms, which are extensions of the rank-1 updating method, achieve the updating using approximately 2(k + n) compound disjoint Givens rotations (CDGRs) with elements annihilated by rotations in adjacent planes. Block generalization of the serial rank-1 algorithms are also presented. The algorithms are rich in level 3 BLAS operations, making them suitable for implementation on large scale parallel systems. The performance of some of the algorithms on a 2-D SIMD (single instruction stream--multiple instruction stream) array processor is discussed.

SP2 system architecture

Scalable parallel systems are increasingly being used today to address existing and emerging application areas that require performance levels significantly beyond what symmetric multiprocessors are capable of providing. These areas include traditional technical computing applications, commercial computing applications such as decision support and transaction processing, and emerging areas such as “grand challenge” applications, digital libraries, and video production and distribution. The IBM SP2™ is a general-purpose scalable parallel system designed to address a wide range of these applications. This paper gives an overview of the architecture and structure of SP2, discusses the rationale for the significant system design decisions that were made, indicates the extent to which key objectives were met, and identifies future system challenges and advanced technology development areas.

Vector transfer by self-tested self-synchronization for parallel systems

Communications between processing elements (PEs)in very large scale parallel systems become more challenging as the function and speed of the PEs improve continuously. Clocked I/O ports may malfunction if data read failure occurs due to clock skew. There are many drawbacks in global clock distribution utilized to reduce the clock skew. This paper addresses a self-tested self-synchronization (STSS) method for vector transfer between PEs. A test signal is added to remove the data read failure. The advantages of this method are: very high data throughput, less power consumption in clock distribution, no constraints on clock skew and system scale, easy in design, less latency. A failure zone concept is used to characterize the behavior of storage elements. By using a jitter injected test signal, a robust vector transfer between PEs with arbitrary clock phases is achieved and the headache problem of the global synchronization is avoided.

High-level programming of massively parallel computers based on shared virtual memory

Highly parallel machines needed to solve compute-intensive scientific applications are based on the distribution of physical memory across the compute nodes. The drawback of such systems is the necessity to write applications in the message passing programming model. Therefore, a lot of research is going on in higher-level programming models and supportive hardware, operating system techniques, languages. The research direction outlined in this article is based on shared virtual memory systems, i.e., scalable parallel systems with a global address space which support an adaptive mapping of global addresses to physical memories. We introduce programming concepts and program optimizations for SVM systems in the context of the SVM-Fortran programming environment which is based on a shared virtual memory system implemented on Intel Paragon. The performance results for real applications proved that this environment enables users to obtain a similar or better performance than by programming in HPF.

Optimal and Near–Optimal Solutions for Hard Compilation Problems

An optimizing compiler typically uses multiple program representations at different levels of program and performance abstractions in order to be able to perform transformations that – at least in the majority of cases – will lead to an overall improvement in program performance. The complexities of the program and performance abstractions used to formulate compiler optimization problems have to match the complexities of the high–level programming model and of the underlying target system. Scalable parallel systems typically have multi–level memory hierarchies and able to exploit coarse–grain and fine–grain parallelism. Most likely, future systems will have even deeper memory hierarchies and more granularities of parallelism. As a result, future compiler optimizations will have to use more and more complex, multi–level computation and performance models in order to keep up with the complexities of their future target systems. Most of the optimization problems encountered in highly optimizing compilers are already NP–hard, and there is little hope that most newly encountered optimization formulations will not be at least NP–hard as well. To face this "complexity crisis", new methods are needed to evaluate the benefits of a compiler optimization formulation. A crucial step in this evaluation process is to compute the optimal solution of the formulation. Using ad–hoc methods to compute optimal solutions to NP–complete problems may be prohibitively expensive. Recent improvements in mixed integer and 0–1 integer programming suggest that this technology may provide the key to efficient, optimal and near–optimal solutions to NP–complete compiler optimization problems. In fact, early results indicate that integer programming formulations may be efficient enough to be included in not only evaluation prototypes, but in production programming environments or even production compilers. This paper discusses the potential benefits of integer programming as a tool to deal with NP–complete compiler optimization formulations in compilers and programming environments.

ADAPTIVE PARALLEL MULTIGRID SOLUTION OF 2D INCOMPRESSIBLE NAVIER-STOKES EQUATIONS

In this paper an adaptive parallel multigrid method and an application example for the 2D incompressible Navier–Stokes equations are described. The strategy of the adaptivity in the sense of local grid refinement in the multigrid context is the multilevel adaptive technique (MLAT) suggested by Brandt. The parallelization of this method on scalable parallel systems is based on the portable communication library CLIC and the message-passing standards: PARMACS, PVM and MPI. The specific problem considered in this work is a two-dimensional hole pressure problem in which a Poiseuille channel flow is disturbed by a cavity on one side of the channel. Near geometric singularities a very fine grid is needed for obtaining an accurate solution of the pressure value. Two important issues of the efficiency of adaptive parallel multigrid algorithms, namely the data redistribution strategy and the refinement criterion, are discussed here. For approximate dynamic load balancing, new data in the adaptive steps are redistributed into distributed memories in different processors of the parallel system by block remapping. Among several refinement criteria tested in this work, the most suitable one for the specific problem is that based on finite-element residuals from the point of view of self-adaptivity and computational efficiency, since it is a kind of error indicator and can stop refinement algorithms in a natural way for a given tolerance. Comparisons between different global grids without and with local refinement have shown the advantages of the self-adaptive technique, as this can save computer memory and speed up the computing time several times without impairing the numerical accuracy. © 1997 By John Wiley & Sons, Ltd. Int. J. Numer. Methods Fluids 24, 875–892, 1997.

ADAPTIVE PARALLEL MULTIGRID SOLUTION OF 2D INCOMPRESSIBLE NAVIER–STOKES EQUATIONS

In this paper an adaptive parallel multigrid method and an application example for the 2D incompressible Navier–Stokes equations are described. The strategy of the adaptivity in the sense of local grid refinement in the multigrid context is the multilevel adaptive technique (MLAT) suggested by Brandt. The parallelization of this method on scalable parallel systems is based on the portable communication library CLIC and the message-passing standards: PARMACS, PVM and MPI. The specific problem considered in this work is a two-dimensional hole pressure problem in which a Poiseuille channel flow is disturbed by a cavity on one side of the channel. Near geometric singularities a very fine grid is needed for obtaining an accurate solution of the pressure value. Two important issues of the efficiency of adaptive parallel multigrid algorithms, namely the data redistribution strategy and the refinement criterion, are discussed here. For approximate dynamic load balancing, new data in the adaptive steps are redistributed into distributed memories in different processors of the parallel system by block remapping. Among several refinement criteria tested in this work, the most suitable one for the specific problem is that based on finite-element residuals from the point of view of self-adaptivity and computational efficiency, since it is a kind of error indicator and can stop refinement algorithms in a natural way for a given tolerance. Comparisons between different global grids without and with local refinement have shown the advantages of the self-adaptive technique, as this can save computer memory and speed up the computing time several times without impairing the numerical accuracy. © 1997 By John Wiley & Sons, Ltd. Int. J. Numer. Methods Fluids 24, 875–892, 1997.

Virtual reality and parallel systems performance analysis

Scalable parallel systems are becoming the standard architecture for high-performance computing. However, achieving close to peak performance requires careful attention to a plethora of system details. Not only do hundreds of processors interact on a microsecond time scale, but also the space of possible performance optimizations is large, complex, and highly sensitive to both application behavior and system software. If the event frequency is high and the number of processors is large (in the hundreds), the aggregate data rate can be many megabytes/second. Moreover, for a fixed application problem size, both processor-interaction frequency and performance data volume can grow superlinearly with the number of processors. Finally, the relations of specific performance metrics to application performance can vary widely across applications and parallel architectures. To understand these relations while managing potentially large volumes of dynamic performance data, we have developed a data-immersive virtual environment, called Avatar, which explores performance data and provides real-time adaptive control of application behavior.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

SP2 system architecture

Scalable parallel systems are increasingly being used today to address existing and emerging application areas that require performance levels significantly beyond what symmetric multiprocessors are capable of providing. These areas include traditional technical computing applications, commercial computing applications such as decision support and transaction processing, and emerging areas such as “grand challenge” applications, digital libraries, and video production and distribution. The IBM SP2™ is a general-purpose scalable parallel system designed to address a wide range of these applications. This paper gives an overview of the architecture and structure of SP2, discusses the rationale for the significant system design decisions that were made, indicates the extent to which key objectives were met, and identifies future system challenges and advanced technology development areas.