Large-scale Parallel Programs Research Articles

MFFT: A GPU Accelerated Highly Efficient Mixed-Precision Large-Scale FFT Framework

Fast Fourier transform (FFT) is widely used in computing applications in large-scale parallel programs, and data communication is the main performance bottleneck of FFT and seriously affects its parallel efficiency. To tackle this problem, we propose a new large-scale FFT framework, MFFT, which optimizes parallel FFT with a new mixed-precision optimization technique, adopting the “high precision computation, low precision communication” strategy. To enable “low precision communication”, we propose a shared-exponent floating-point number compression technique, which reduces the volume of data communication, while maintaining higher accuracy. In addition, we apply a two-phase normalization technique to further reduce the round-off error. Based on the mixed-precision MFFT framework, we apply several optimization techniques to improve the performance, such as streaming of GPU kernels, MPI message combination, kernel optimization, and memory optimization. We evaluate MFFT on a system with 4,096 GPUs. The results show that shared-exponent MFFT is 1.23 × faster than that of double-precision MFFT on average, and double-precision MFFT achieves performance 3.53× and 9.48× on average higher than open source library 2Decomp&FFT (CPU-based version) and heFFTe (AMD GPU-based version), respectively. The parallel efficiency of double-precision MFFT increased from 53.2% to 78.1% compared with 2Decomp&FFT, and shared-exponent MFFT further increases the parallel efficiency to 83.8%.

Optimisation of the finite-difference scheme based on an improved PSO algorithm for elastic modelling

The finite-difference (FD) scheme is extensively applied in seismic modelling, imaging and inversion due to its advantages of large-scale parallel computing and programming. However, numerical dispersion caused by using a difference operator in substitution for the differential operator is non-negligible, which reduces the accuracy of the modelling and can lead to some misinterpretations. In addition, the computing resources required by the FD scheme is highly demanding when dealing with large models, which limits its applicability. In this paper, a new optimised FD scheme is proposed, which is based on an improved particle swarm optimisation (PSO) algorithm. We improve the conventional PSO algorithm by introducing strategies related to local learning and global learning, which contribute to accelerating the convergence rate and effectively avoiding getting trapped in local extrema. Then, the improved PSO algorithm is used to improve the conventional FD scheme. Dispersion analysis and numerical modelling demonstrate that the low-order optimised FD scheme can achieve higher accuracy than a high-order conventional operator. Compared with the conventional FD scheme and a FD scheme based on the Remez exchange algorithm, the optimised FD scheme based on the improved PSO algorithm can more efficiently suppress numerical dispersion and increase computational efficiency.

ELS: Emulation system for debugging and tuning large-scale parallel programs on small clusters

Continuous scaling-up of high-performance computing systems has brought challenges to the debugging and tuning of large-scale parallel programs. Firstly, to locate bugs in a program or tune its performance, programmer often needs to execute the program in a specified scale repeatedly, which consumes massive resources; secondly, due to the extensively used job scheduling systems, programmers can only submit their programs as jobs and cannot interact with them, which restricts debugging efficiency and flexibility. To address these challenges, this paper proposes an emulation system that supports debugging and tuning of large-scale parallel programs by executing parallel programs in the desired scale on a small cluster. The program is firstly executed in the desired scale on the target HPC system to record necessary information; then, programmers can choose and re-execute a subset of processes of the program repeatedly on a small cluster, during which the emulation system controls the execution of the processes, and programmers can debug their programs by attaching tools to the selected processes. Moreover, our system supports popular CPU+GPU heterogeneous architecture. The system is evaluated on a small cluster, while a 1000-node system is used as the target HPC system; experimental results demonstrate the accuracy and efficiency of emulation-execution.

A Lightweight and Flexible Tool for Distinguishing Between Hardware Malfunctions and Program Bugs in Debugging Large-Scale Programs

In this paper, we propose a new technique to distinguish the reason for program failure between hardware malfunctions and program bugs, which mitigates the impact of shorter mean time between failures to the debugging process on the future exa-scale supercomputers and improves the productivity of debugging large-scale parallel programs. Our technique detects program failures by observing the abnormal message passing behaviors with distributed monitors and leverages event-driven mechanism to trigger global status checking among different node groups concurrently. Besides, both coarse-grained execution snapshots and fine-grained failure events can be provided for further failure diagnosis and bug analysis. We implement this technique as a user-space library named failure cause resolver (FCR). Experimental results on the Tianhe-2 supercomputer demonstrate that the latency of FCR for failure detection is acceptable with negligible overhead. In addition, FCR does not require administrative privilege and can be easily integrated into existing large-scale parallel programs.

Critical-blame analysis for OpenMP 4.0 offloading on Intel Xeon Phi

Critical-path detection in OpenMP 4.0 programs with offloaded code.Detection and quantification of load imbalances and their cause in OpenMP 4.0 codes.Implementation in the open-source tool infrastructure Score-P.Validation and evaluation with modified SPEC ACCEL benchmarks. Recent supercomputers rated in the TOP 500 list increasingly utilize accelerator or co-processor devices to improve performance and energy efficiency. Since version 4.0 of the specification OpenMP addresses this heterogeneity in computing with the target directives, which enable programmers to offload portions of the code to massively-parallel target devices. Due to this new complexity in hardware and software design, performance optimization of large-scale parallel programs becomes more and more challenging. As manual performance analysis is getting infeasible for complex high performance computing (HPC) codes, we propose an approach to automatically detect bottlenecks such as load imbalances in heterogeneous OpenMP applications. We developed a method to perform critical-path and root-cause analysis for the OpenMP4.0offloading model and integrated it into the tool CASITA. The post-mortem analysis is based on execution traces that are generated with an implementation of the evolving OpenMP Tools Interface into the measurement system Score-P. To validate the implementation of our method we ported several existing codes to OpenMP4.0, executed them with an Intel Xeon Phi as the target device and analyzed the resulting trace files.

Online MPI Trace Compression Using Event Flow Graphs and Wavelets

Performance analysis of scientific parallel applications is essential to use High Performance Computing (HPC) infrastructures efficiently. Nevertheless, collecting detailed data of large-scale parallel programs and long-running applications is infeasible due to the huge amount of performance information generated. Even though there are no technological constraints in storing Terabytes of performance data, the constant flushing of such data to disk introduces a massive overhead into the application that makes the performance measurements worthless. This paper explores the use of Event flow graphs together with wavelet analysis and EZW-encoding to provide MPI event traces that are orders of magnitude smaller while preserving accurate information on timestamped events. Our mechanism compresses the performance data online while the application runs, thus, reducing the pressure put on the I/O system due to buffer flushing. As a result, we achieve lower application perturbation, reduced performance data output, and the possibility to monitor longer application runs.

Combing the Communication Hairball: Visualizing Parallel Execution Traces using Logical Time.

With the continuous rise in complexity of modern supercomputers, optimizing the performance of large-scale parallel programs is becoming increasingly challenging. Simultaneously, the growth in scale magnifies the impact of even minor inefficiencies--potentially millions of compute hours and megawatts in power consumption can be wasted on avoidable mistakes or sub-optimal algorithms. This makes performance analysis and optimization critical elements in the software development process. One of the most common forms of performance analysis is to study execution traces, which record a history of per-process events and interprocess messages in a parallel application. Trace visualizations allow users to browse this event history and search for insights into the observed performance behavior. However, current visualizations are difficult to understand even for small process counts and do not scale gracefully beyond a few hundred processes. Organizing events in time leads to a virtually unintelligible conglomerate of interleaved events and moderately high process counts overtax even the largest display. As an alternative, we present a new trace visualization approach based on transforming the event history into logical time inferred directly from happened-before relationships. This emphasizes the code's structural behavior, which is much more familiar to the application developer. The original timing data, or other information, is then encoded through color, leading to a more intuitive visualization. Furthermore, we use the discrete nature of logical timelines to cluster processes according to their local behavior leading to a scalable visualization of even long traces on large process counts. We demonstrate our system using two case studies on large-scale parallel codes.

TBBench: A Micro-Benchmark Suite for Intel Threading Building Blocks

Task-based programming is becoming the state-of-the-art method of choice for extracting the desired performance from multi-core chips. It expresses a program in terms of lightweight logical tasks rather than heavyweight threads. Intel Threading Building Blocks (TBB) is a task-based parallel programming paradigm for multi-core processors. The performance gain of this paradigm depends to a great extent on the efficiency of its parallel constructs. The parallel overheads incurred by parallel constructs determine the ability for creating large-scale parallel programs, especially in the case of fine-grain parallelism. This paper presents a study of TBB parallelization overheads. For this purpose, a TBB micro-benchmarks suite called TBBench has been developed. We use TBBench to evaluate the parallelization overheads of TBB on different multi-core machines and different compilers. We report in detail in this paper on the relative overheads and analyze the running results.

Determine energy-saving potential in wait-states of large-scale parallel programs

Energy consumption is one of the major topics in high performance computing (HPC) in the last years. However, little effort is put into energy analysis by developers of HPC applications.We present our approach of combined performance and energy analysis using the performance analysis tool-set Scalasca. Scalascas parallel wait-state analysis is extended by a calculation of the energy-saving potential if a lower power-state can be used.

Scalable communication protocols for dynamic sparse data exchange

Many large-scale parallel programs follow a bulk synchronous parallel (BSP) structure with distinct computation and communication phases. Although the communication phase in such programs may involve all (or large numbers) of the participating processes, the actual communication operations are usually sparse in nature. As a result, communication phases are typically expressed explicitly using point-to-point communication operations or collective operations. We define the dynamic sparse data-exchange (DSDE) problem and derive bounds in the well known LogGP model. While current approaches work well with static applications, they run into limitations as modern applications grow in scale, and as the problems that are being solved become increasingly irregular and dynamic. To enable the compact and efficient expression of the communication phase, we develop suitable sparse communication protocols for irregular applications at large scale. We discuss different irregular applications and show the sparsity in the communication for real-world input data. We discuss the time and memory complexity of commonly used protocols for the DSDE problem and develop NBX --a novel fast algorithm with constant memory overhead for solving it. Algorithm NBX improves the runtime of a sparse data-exchange among 8,192 processors on BlueGene/P by a factor of 5.6. In an application study, we show improvements of up to a factor of 28.9 for a parallel breadth first search on 8,192 BlueGene/P processors.

Error detection in large-scale parallel programs with long runtimes

Error detection is an important activity of program development, which is applied to detect incorrect computations or runtime failures of software. The costs of debugging are strongly related to the complexity and the scale of the investigated programs. Both characteristics are especially cumbersome for large-scale parallel programs with long runtimes, which are quite common in computational science and engineering (CSE) applications. A solution is offered by a combination of techniques using the event graph model as a representation of parallel program behaviour. With process isolation, a subset of the original number of processes can be investigated, while the absent processes are simulated by the debugging system. With checkpointing, an arbitrary temporal section of a program’s runtime can be extracted for exhaustive analysis without the need to restart the program from the beginning. Additional benefits of the event graph are support of equivalent execution of nondeterministic programs, as well as a comprehensible visualisation as a space–time diagram.

The Paradyn parallel performance measurement tool

Paradyn is a tool for measuring the performance of large-scale parallel programs. Our goal in designing a new performance tool was to provide detailed, flexible performance information without incurring the space (and time) overhead typically associated with trace-based tools. Paradyn achieves this goal by dynamically instrumenting the application and automatically controlling this instrumentation in search of performance problems. Dynamic instrumentation lets us defer insertion until the moment it is needed (and remove it when it is no longer needed); Paradyn's Performance Consultant decides when and where to insert instrumentation.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

“Topologies”—distributed objects on multicomputers

Application programs written for large-scale multicomputers with interconnection structures known to the programmer (e.g., hypercubes or meshes) use complex communication structures for connecting the applications' parallel tasks. Such structures implement a wide variety of functions, including the exchange of data or control information relevant to the task computations and/or the communications required for task synchronization, message forwarding/filtering under program control, and so on.Topologyis a programming and operating system construct that allows programmers to describe and efficiently implement such functionality as distributed objects with well-defined operational interfaces. As with abstract data types, topologies may be reused by any application desiring their functionality. However, in contrast to other research in parallel or distributed object-based operating systems, internally, a topology may be an entirely distributed implementation of the object's functionality, consisting of a communication graph and type-specific computations, which are triggered by messages traversing the graph. Sample computations may perform additions or minimizations of the values traversing a topology, thereby computing a global sum or minimum. Similarly, computations may concatenate or filter messages in order to implement program monitoring, I/O, file storage, or virtual terminal services.Topologies are implemented as an extension of the Intel iPSC hypercube's operating system kernel and have been used with several, large-scale parallel application programs.

An integrated approach to parallel program debugging and performance analysis onlarge-scale multiprocessors

One of the most serious problems in the development cycle of large-scale parallel programs is the lack of tools for debugging and performance analysis. We are addressing the problem on large-scale, shared-memory multiprocessors. We have developed a dynamic, fine-grain characterization of parallel program executions based on a partial order of accesses to shared objects. This characterization forms the basis for an extensible, integrated toolkit for parallel program debugging and performance analysis. Tools in the toolkit include an interactive debugger, a graphical execution browser, performance analysis packages, and a programmable interface for user queries. Using our integrated toolkit approach, programmers can interactively analyze the behavior of parallel programs in much the same way interactive debuggers and profilers are used to analyze the behavior of sequential programs.

Large-scale parallel programming: experience with BBN butterfly parallel processor

For three years, members of the Computer Science Department at the University of Rochester have used a collection of BBN Butterfly™ Parallel Processors to conduct research in parallel systems and applications. For most of that time, Rochester's 128-node machine has had the distinction of being the largest shared-memory multiprocessor in the world. In the course of our work with the Butterfly we have ported three compilers, developed five major and several minor library packages, built two different operating systems, and implemented dozens of applications. Our experience clearly demonstrates the practicality of large-scale shared-memory multiprocessors, with non-uniform memory access times. It also demonstrates that the problems inherent in programming such machines are far from adequately solved. Both locality and Amdahl's law become increasingly important with a very large number of nodes. The availability of multiple programming models is also a concern; truly general-purpose parallel computing will require the development of environments that allow programs written under different models to coexist and interact. Most important, there is a continuing need for high-quality programming tools; widespread acceptance of parallel machines will require the development of programming environments comparable to those available on sequential computers.