MPI Communication Research Articles

Reducing Inter-Process Communication Overhead in Parallel Sparse Matrix-Matrix Multiplication

Parallel sparse matrix-matrix multiplication algorithms (PSpGEMM) spend most of their running time on inter-process communication. In the case of distributed matrix-matrix multiplications, much of this time is spent on interchanging the partial results that are needed to calculate the final product matrix. This overhead can be reduced with a one-dimensional distributed algorithm for parallel sparse matrix-matrix multiplication that uses a novel accumulation pattern based on the logarithmic complexity of the number of processors (i.e., where is the number of processors). This algorithm's MPI communication overhead and execution time were evaluated on an HPC cluster, using randomly generated sparse matrices with dimensions up to one million by one million. The results showed a reduction of inter-process communication overhead for matrices with larger dimensions compared to another one dimensional parallel algorithm that takes run-time complexity for accumulating the results.

Scalable communication event tracing via clustering

Communication traces help developers of high-performance computing (HPC) applications understand and improve their codes. When run on large-scale HPC facilities, the scalability of tracing tools becomes a challenge. To address this problem, traces can be clustered into groups of processes that exhibit similar behavior. Instead of collecting trace information of each individual node, it then suffices to collect a trace of a small set of representative nodes, namely one per cluster. However, clustering algorithms themselves need to have low overhead, be scalable, and adapt to application characteristics. We devised an adaptive clustering algorithm for large-scale applications called ACURDION that traces the MPI communication of code with O(log P) time complexity. First, ACURDION identifies the parameters that differ across processes by using a logarithmic algorithm called Adaptive Signature Building. Second, it clusters the processes based on those parameters. Experiments show that collecting traces of just nine nodes/clusters suffices to capture the communication behavior of all nodes for a wide set of HPC benchmarks codes while retaining sufficient accuracy of trace events and parameters. In summary, ACURDION improves trace scalability and automation over prior approaches.

GPU-accelerated red blood cells simulations with transport dissipative particle dynamics

Mesoscopic numerical simulations provide a unique approach for the quantification of the chemical influences on red blood cell functionalities. The transport Dissipative Particle Dynamics (tDPD) method can lead to such effective multiscale simulations due to its ability to simultaneously capture mesoscopic advection, diffusion, and reaction. In this paper, we present a GPU-accelerated red blood cell simulation package based on a tDPD adaptation of our red blood cell model, which can correctly recover the cell membrane viscosity, elasticity, bending stiffness, and cross-membrane chemical transport. The package essentially processes all computational workloads in parallel by GPU, and it incorporates multi-stream scheduling and non-blocking MPI communications to improve inter-node scalability. Our code is validated for accuracy and compared against the CPU counterpart for speed. Strong scaling and weak scaling are also presented to characterize scalability. We observe a speedup of 10.1 on one GPU over all 16 cores within a single node, and a weak scaling efficiency of 91% across 256 nodes. The program enables quick-turnaround and high-throughput numerical simulations for investigating chemical-driven red blood cell phenomena and disorders. Program summaryProgram Title: USERMESO 2.0Program Files doi:http://dx.doi.org/10.17632/89849t3ngk.1Licensing provisions: GNU General Public License, Version 3Programming language: C/C++, CUDA C/C++, MPI.Nature of problem: Particle-based simulation of a red blood cell suspension with chemical transport property.Solution method: Each red blood cell is represented by a 3-D triangular mesh with bonded potential under area and volume constraints. The solvent is approximated with coarse-grained particles. The time evolution of the system is integrated using Velocity-Verlet algorithm.Restrictions: The code is compatible with NVIDIA GPGPUs with compute capability 3.0 and above.Unusual features: The code is implemented on GPGPUs with significantly improved speed.Additional Comments: Github repository link https://github.com/AnselGitAccount/USERMESO-2.0

Leveraging the accelerated processing units for seismic imaging: A performance and power efficiency comparison against CPUs and GPUs

Oil and gas companies rely on high performance computing to process seismic imaging algorithms such as reverse time migration. Graphics processing units are used to accelerate reverse time migration, but these deployments suffer from limitations such as the lack of high graphics processing unit memory capacity, frequent CPU-GPU communications that may be bottlenecked by the PCI bus transfer rate, and high power consumptions. Recently, AMD has launched the Accelerated Processing Unit (APU): a processor that merges a CPU and a graphics processing unit on the same die featuring a unified CPU-GPU memory. In this paper, we explore how efficiently may the APU be applicable to reverse time migration. Using OpenCL (along with MPI and OpenMP), a CPU/APU/GPU comparative study is conducted on a single node for the 3D acoustic reverse time migration, and then extended on up to 16 nodes. We show the relevance of overlapping the I/O and MPI communications with the computations for the APU and graphics processing unit clusters, that performance results of APUs range between those of CPUs and those of graphics processing units, and that the APU power efficiency is greater than or equal to the graphics processing unit one.

Modelling parallel overhead from simple run-time records

A number of scientific applications run on current HPC systems would benefit from an approximate assessment of parallel overhead. In many instances a quick and simple method to obtain a general overview on the subject is regarded useful auxiliary information by the routine HPC user. Here we present such a method using just execution times for increasing numbers of parallel processing cores. We start out with several common scientific applications and measure the fraction of time spent in MPI communication. Forming the ratio of MPI time to overall execution time we obtain a smooth curve that can be parameterized by only two constants. We then use this two-parameter expression and extend Amdahl’s theorem with a new term representing parallel overhead in general. Fitting the original data set with this extended Amdahl expression yields an estimate for the parallel overhead closely matching the MPI time determined previously.

LU factorization on heterogeneous systems: an energy-efficient approach towards high performance

Dense lower–upper (LU) factorization (hereafter referred to as LU) is a critical kernel that is widely used to solve dense linear algebra problems. Hybrid LU algorithms have been well designed to exploit the full capacity of heterogeneous systems. However, existing heterogeneous implementations are typically CPU-centric, which rely highly on CPU cores and suffer from a large amount of data transfers via the PCIe bus, and thus reduce the overall energy efficiency of the entire computer system. In this paper, we provide a coprocessor-resident implementation of LU for a heterogeneous platform to improve energy efficiency by relieving the CPUs from performing heavy load computations and avoiding excessive data transfers via PCIe. To maintain the performance, we conduct optimizations to pipeline the CPU computation, coprocessor computation, MPI communication, and PCIe transfer between the CPUs and coprocessors. The experiments on the Tianhe-2 supercomputer show that our LU implementation can compete with the highly optimized Intel MKL implementation in performance and overcome the limitations of energy efficiency.

A study of the influence of VM allocation policies on MPI Bcast and MPI Exchange latency in cloud

This article analyzes the influence of VM allocation policies in cloud on the latency of MPI communications. Two popular MPI collectives, Bcast and Exchange, have been analyzed over two types of infrastructures: a cloud based on Apache CloudStack, employing commodity hardware, and a non-virtualized cluster, also employing commodity hardware. The main goal of this article is analyzing the effects on the MPI latency that introduce the allocation police employed by the cloud platform to deploy the Virtual Machines. Two opposite allocation schemes have been considered: concentrated, in which the Virtual Machines are allocated in the same host, and scattered, in which the Virtual Machines are spread using different hosts. The MPI communications have been analyzed for message sizes ranging from 1 byte to 4096 KiB. The results show that the cloud layer introduces a considerable overhead in the latency of Bcast and Exchange communications. However, if the cloud is employed to execute MPI applications, the concentrated allocation scheme is suitable for messages of less than 4 KiB, whereas for larger messages the scattered allocation scheme is better.

Simulator test suite for evaluating performance of multithreaded Message passing interface execution on SUN cluster

The Message Passing Interface Standard (MPI) is a message passing library standard based on the consensus of the MPI Forum, which has over 40 participating organizations, including vendors, researchers, software library developers, and users. The goal of the Message Passing Interface is to establish a portable, efficient, and flexible standard for message passing that will be widely used for writing message passing programs. As such, MPI is the first standardized, vendor independent, message-passing library. The advantages of developing message passing software using MPI closely match the design goals of portability, efficiency, and flexibility. MPI is not an IEEE or ISO standard, but has, in fact, become the "industry standard" for writing message passing programs on HPC platforms. As parallel systems are commonly being built out of increasingly large multicore chips, Application programmers are exploring the use of hybrid programming models combining MPI across nodes and multithreading within a node. Many MPI implementations, however, are just starting to support, multithreaded MPI communication, often focusing on correctness First and performance later. The MPI implementation defines functions that can be used for initializing the thread environment. It is not required that all MPI implementations fulfill all the requirements which are All MPI calls are thread-safe and Blocking MPI calls. MPI process is a process that may be multi-threaded. Each thread can issue MPI calls. The MPI Standard, however, requires only that no MPI call in one thread block MPI calls in other threads; it makes no performance guarantees. In this paper propose a test suite to measure the performance. The test has seven benchmarks which are overhead of MPI_thread_multiple level for thread safety, concurrent bandwidth, concurrent latency, concurrent short-long messages, communication/computation overlap, concurrent collective and concurrent collective and computation.

High performance Python for direct numerical simulations of turbulent flows

Direct Numerical Simulations (DNS) of the Navier Stokes equations is an invaluable research tool in fluid dynamics. Still, there are few publicly available research codes and, due to the heavy number crunching implied, available codes are usually written in low-level languages such as C/C++ or Fortran. In this paper we describe a pure scientific Python pseudo-spectral DNS code that nearly matches the performance of C++ for thousands of processors and billions of unknowns. We also describe a version optimized through Cython, that is found to match the speed of C++. The solvers are written from scratch in Python, both the mesh, the MPI domain decomposition, and the temporal integrators. The solvers have been verified and benchmarked on the Shaheen supercomputer at the KAUST supercomputing laboratory, and we are able to show very good scaling up to several thousand cores.A very important part of the implementation is the mesh decomposition (we implement both slab and pencil decompositions) and 3D parallel Fast Fourier Transforms (FFT). The mesh decomposition and FFT routines have been implemented in Python using serial FFT routines (either NumPy, pyFFTW or any other serial FFT module), NumPy array manipulations and with MPI communications handled by MPI for Python (mpi4py). We show how we are able to execute a 3D parallel FFT in Python for a slab mesh decomposition using 4 lines of compact Python code, for which the parallel performance on Shaheen is found to be slightly better than similar routines provided through the FFTW library. For a pencil mesh decomposition 7 lines of code is required to execute a transform.

A Hybrid Parallel Delaunay Image-to-mesh Conversion Algorithm Scalable on Distributed-memory Clusters

In this paper, we present a scalable three dimensional hybrid MPI+Threads parallel Delaunay image-to-mesh conversion algorithm. A nested master-worker communication model for parallel mesh generation is implemented which simultaneously explores process-level parallelization and thread-level parallelization: inter-node communication using MPI and inter-core communication inside one node using threads. In order to overlap the communication (task request and data movement) and computation (parallel mesh refinement), the inter-node MPI communication and intra-node local mesh refinement is separated. The master thread that initializes the MPI environment is in charge of the inter-node MPI communication while the worker threads of each process are only responsible for the local mesh refinement within the node. We conducted a set of experiments to test the performance of the algorithm on Turing, a distributed memory cluster at Old Dominion University High Performance Computing Center and observed that the granularity of coarse level data decomposition, which affects the coarse level concurrency, has a significant influence on the performance of the algorithm. With the proper value of granularity, the algorithm expresses impressive performance potential and is scalable to 30 distributed memory compute nodes with 20 cores each (the maximum number of nodes available for us in the experiments).

623 Tflop/s HPCG run on Tianhe-2: Leveraging millions of hybrid cores

In this article, we present a new hybrid algorithm to enable and scale the high-performance conjugate gradients (HPCG) benchmark on large-scale heterogeneous systems such as the Tianhe-2. Based on an inner–outer subdomain partitioning strategy, the data distribution between host and device can be balanced adaptively. The overhead of data movement from both the MPI communication and the PCI-E transfer can be significantly reduced by carefully rearranging and fusing operations. A variety of parallelization and optimization techniques for performance-critical kernels are exploited and analyzed to maximize the performance gain on both host and device. We carry out experiments on both a small heterogeneous computer and the world’s largest one, the Tianhe-2. On the small system, a thorough comparison and analysis has been presented to select from different optimization choices. On Tianhe-2, the optimized implementation scales to the full-system level of 3.12 million heterogeneous cores, with an aggregated performance of 623 Tflop/s and a parallel efficiency of 81.2%.

Real-time and real-space program tuned in K-computer

We have transported the code from Earth Simulator (ES) to K-Computer (KC) keeping a scalability. The base code achieved remarkable linear scalability and significant parallelization efficiency in EC before. The reason of success is that, taking into account the architectural differences between ES and KC, we improve the program step by step. Actually, the program showed a 37% improvement in the total performance with the implementations such as hybrid OpenMP and MPI parallelization, and the optimal rank mapping, and the MPI communication concealment, including removing the unnecessary processes such as zero clear and data copy.

Leveraging OpenCoarrays to Support Coarray Fortran on IBM Power8E

We include a few comparisons between the performance a competent application developer would attain with two-sided MPI communication and the performance achieved with the newer MPI 3.0 one-sided communication that OpenCoarrays generates from CAF programs. We also demonstrate the OpenCoarrays support for the parallel collective subroutines that have been proposed for the upcoming Fortran 2015 standard.

Hybrid parallelization of the LIGGGHTS open-source DEM code

This work presents our efforts to implement an MPI/OpenMP hybrid parallelization of the LIGGGHTS open-source software package for Discrete Element Methods (DEM). We outline the problems encountered and the solutions implemented to achieve scalable performance using both parallelization models. Three case studies, including two real-world applications with up to 1.5 million particles, were evaluated and demonstrate the practicality of this approach. In these examples, better load balancing and reduced MPI communication led to speed increases of up to 44% compared to MPI-only simulations.

τ-Lop: Modeling performance of shared memory MPI

We present a performance model for representing and predicting cost of parallel algorithms.Model goal is to help in the design and optimization of parallel collective algorithms.It is applied to collective algorithms in mainstream MPI implementations in shared memory.Model is compared to other well known and established models. Formal modeling of the cost of MPI primitives allows a machine independent representation, comparison and performance analysis of their underlying algorithms. Current accepted methods are all the off-springs of LogP, conceived to model the cost of inter-node point-to-point messages in networks of single-processor machines. As new supercomputers are built upon cheap commodity boards with a growing number of cores accessing hierarchical memories, intra-node communication becomes progressively more relevant. Techniques for shared memory communication, such as message segmentation and collectives, not based on point-to-point operations, are substantively different from their inter-node counterparts. This paper unveils the reasons for the poor fit of LogGP and the most recent models in this domain, log n P and m log n P , and proposes a new model named ? -Lop, rooted on them, but addressing the challenge of accurately modeling shared memory MPI communications. Broadcast algorithms of mainstream MPI implementations, MPICH and Open MPI, are modeled and analyzed.

The Particle Accelerator Simulation Code PyORBIT

Abstract The particle accelerator simulation code PyORBIT is presented. The structure, implementation, history, parallel and simulation capabilities, and future development of the code are discussed. The PyORBIT code is a new implementation and extension of algorithms of the original ORBIT code that was developed for the Spallation Neutron Source accelerator at the Oak Ridge National Laboratory. The PyORBIT code has a two level structure. The upper level uses the Python programming language to control the flow of intensive calculations performed by the lower level code implemented in the C + + language. The parallel capabilities are based on MPI communications. The PyORBIT is an open source code accessible to the public through the Google Open Source Projects Hosting Service.

Recent advances in the Message Passing Interface

ion for MPI programs with pointer-based data structures. The main features of DRASync are: it amortizes communication among MPI processes to allow efficient parallel allocation in a global address space; it takes advantage of bulk deallocation and good locality with pointer-based data structures. Finally, DRASync supports ownership semantics of regions by MPI processes akin to reader–writer locks, which makes for a high-level, intuitive synchronization tool in MPI programs, without sacrificing message-passing performance. In ‘‘An Evaluation of MPI Message Rate on Hybrid-Core Processors’’, Barrett et al. analyze the ability of simple and more complex cores to perform MPI matching operations for various scenarios in order to gain insight into how MPI implementations for future hybrid-core processors should be designed. The authors compare throughput-oriented cores to University Carlos III of Madrid, Spain single-thread optimized cores in terms of the ability to perform MPI match processing. The intent of this study is to gain insight into the ability of throughputoriented cores to adequately perform MPI matching and to better understand how MPI implementations on future hybrid-core processors should allocate computing resources to try to optimize performance-critical MPI operations. Collective MPI communications have to be executed in the same order by all processes in their communicator and the same number of times, otherwise it is not conforming to the standard and a deadlock can occur. As soon as the control-flow involving these collective operations becomes more complex, in particular including conditionals on process ranks, ensuring the correction of such code is error-prone. The paper ‘‘PARCOACH: Combining Static and Dynamic Validation of MPI Collective Communications’’ by Saillard et al. proposes a static analysis to detect when such situation occurs, combined with a code transformation that prevents deadlocking. They show with several benchmarks the small impact on performance and the ease of integration of their techniques in the development process. In ‘‘Extreme-scale Computing Services Over MPI: Experiences, Observations and Features Proposal for Next Generation Message Passing Interface’’ by Zounmevo et al., the authors present their experiences in using MPI as a network transport for a large-scale distributed storage system. The authors discuss the features of MPI that facilitate adoption as well as aspects which require various workarounds. Based on use cases, the authors derive a wish-list for both MPI implementations and the MPI forum to facilitate the adoption of MPI large-scale persistent services. Finally in the paper ‘‘Optimization of MPI collective operations on the IBM Blue Gene/Q Supercomputer’’, Kumar et al. present scalable algorithms to optimize MPI collective operations by taking advantage of the various features of the Blue Gene/Q torus and collective networks. The authors accelerate summing of network packets with local buffers by the use of the Quad Processing SIMD unit in the Blue Gene/Q cores and executing the sums on multiple communication threads supported by the optimized communication libraries.

PARCOACH: Combining static and dynamic validation of MPI collective communications

Nowadays most scientific applications are parallelized based on MPI communications. Collective MPI communications have to be executed in the same order by all processes in their communicator and the same number of times, otherwise they do not conform to the standard and a deadlock or other undefined behaviour can occur. As soon as the control flow involving these collective operations becomes more complex, in particular including conditionals on process ranks, ensuring the correction of such code is error-prone. We propose in this paper a static analysis to detect when such a situation occurs, combined with a code transformation that prevents deadlocking. We focus on blocking MPI collective operations in single program multiple data applications, assuming MPI calls are not nested in multithreaded regions. We show on several benchmarks the small impact on performance and the ease of integration of our techniques in the development process.

On maximum achievable speeds for field solvers

Purpose – Prompted by the empirical evidence that achievable flow solver speeds for large problems are limited by what appears to be a time of the order of O(0.1) sec/timestep regardless of the number of cores used, the purpose of this paper is to identify why this phenomenon occurs. Design/methodology/approach – A series of timing studies, as well as in-depth analysis of memory and inter-processors transfer requirements were carried out for a typical field solver. The results were analyzed and compared to the expected performance. Findings – The analysis shows that at present flow speeds per core are already limited by the achievable transfer rate to RAM. For smaller domains/larger number of processors, the limiting speed of CFD solvers is given by the MPI communication network. Research limitations/implications – This implies that at present, there is a “limiting useful size” for domains, and that there is a lower limit for the time it takes to update a flowfield. Practical implications – For practical calculations this implies that the time required for running large-scale problems will not decrease markedly once these applications migrate to machines with hundreds of thousands of cores. Originality/value – This is the first time such a finding has been reported in this context.

An Extensible System for Multilevel Automatic Data Partition and Mapping

Automatic data distribution is a key feature to obtain efficient implementations from abstract and portable parallel codes. We present a highly efficient and extensible runtime library that integrates techniques for automatic data partition and mapping. It uses a novel approach to define an abstract interface and a plug-in system to encapsulate different types of regular and irregular techniques, helping to generate codes which are independent of the exact mapping functions selected. Currently, it supports hierarchical tiling of arrays with dense and stride domains, that allows the implementation of both data and task parallelism using a SPMD model. It automatically computes appropriate domain partitions for a selected virtual topology, mapping them to available processors with static or dynamic load-balancing techniques. Our library also allows the construction of reusable communication patterns that efficiently exploit MPI communication capabilities. The use of our library greatly reduces the complexity of data distribution and communication, hiding the details of the underlying architecture. The library can be used as an abstract layer for building generic tiling operations as well. Our experimental results show that the use of this library allows to achieve similar performance as carefully-implemented manual versions for several, well-known parallel kernels and benchmarks in distributed and multicore systems, and substantially reduces programming effort.