MPI Communication Research Articles

Comprehensive Evaluation of the Massively Parallel Direct Simulation Monte Carlo Kernel “Stochastic Parallel Rarefied-Gas Time-Accurate Analyzer” in Rarefied Hypersonic Flows—Part A: Fundamentals

The Direct Simulation Monte Carlo (DSMC) method, introduced by Graeme Bird over five decades ago, has become a crucial statistical particle-based technique for simulating low-density gas flows. Its widespread acceptance stems from rigorous validation against experimental data. This study focuses on four validation test cases known for their complex shock–boundary and shock–shock interactions: (a) a flat plate in hypersonic flow, (b) a Mach 20.2 flow over a 70-degree interplanetary probe, (c) a hypersonic flow around a flared cylinder, and (d) a hypersonic flow around a biconic. Part A of this paper covers the first two cases, while Part B will discuss the remaining cases. These scenarios have been extensively used by researchers to validate prominent parallel DSMC solvers, due to the challenging nature of the flow features involved. The validation requires meticulous selection of simulation parameters, including particle count, grid density, and time steps. This work evaluates the SPARTA (Stochastic Parallel Rarefied-gas Time-Accurate Analyzer) kernel’s accuracy against these test cases, highlighting its parallel processing capability via domain decomposition and MPI communication. This method promises substantial improvements in computational efficiency and accuracy for complex hypersonic vehicle simulations.

Designing and prototyping extensions to the Message Passing Interface in MPICH

As HPC system architectures and the applications running on them continue to evolve, the MPI standard itself must evolve. The trend in current and future HPC systems toward powerful nodes with multiple CPU cores and multiple GPU accelerators makes efficient support for hybrid programming critical for applications to achieve high performance. However, the support for hybrid programming in the MPI standard has not kept up with recent trends. The MPICH implementation of MPI provides a platform for implementing and experimenting with new proposals and extensions to fill this gap and to gain valuable experience and feedback before the MPI Forum can consider them for standardization. In this work, we detail six extensions implemented in MPICH to increase MPI interoperability with other runtimes, with a specific focus on heterogeneous architectures. First, the extension to MPI generalized requests lets applications integrate asynchronous tasks into MPI’s progress engine. Second, the iovec extension to datatypes lets applications use MPI datatypes as a general-purpose data layout API beyond just MPI communications. Third, a new MPI object, MPIX_Stream, can be used by applications to identify execution contexts beyond MPI processes, including threads and GPU streams. MPIX stream communicators can be created to make existing MPI functions thread-aware and GPU-aware, thus providing applications with explicit ways to achieve higher performance. Fourth, MPIX Streams are extended to support the enqueue semantics for offloading MPI communications onto a GPU stream context. Fifth, thread communicators allow MPI communicators to be constructed with individual threads, thus providing a new level of interoperability between MPI and on-node runtimes such as OpenMP. Lastly, we present an extension to invoke MPI progress, which lets users spawn progress threads with fine-grained control to adapt the communication performance to their application designs. We describe the design and implementation of these extensions, provide usage examples, and highlight their expected benefits with performance results.

Simulation and reconstruction for 3D elastic wave using multi-GPU and CUDA-aware MPI

3D finite-difference time-domain numerical simulation and reconstruction based on the domain decomposition technique are essential parts of high-performance computation for reverse-time migration and full-waveform inversion. However, the low GPU utilization in computing for small-sized models and the tremendous memory consumption for large-sized models may result in low computational efficiency and high memory costs. This paper proposes a contiguous memory management (CMM) method and a variable-order wavefield reconstruction (VWR) method. The CMM allocates the memory of many small-sized arrays used for MPI communications on a larger-sized contiguous memory block, which aims to reduce the number of MPI communications between subdomains and improve the communication bandwidth, thus reducing the MPI time overhead and improving the GPU utilization. Meanwhile, the VWR can flexibly set the number of layers of boundary wavefield used for source wavefield reconstruction according to the host memory capacity and accuracy requirements. Since one layer of boundary wavefield could be stored using the VWR, the memory consumption of host memory can be significantly alleviated. Numerical experiments show that GPU utilization in computing for the model with a size of 1213 can be improved from 25% to 90% using the CMM method, and the VWR method can reduce memory consumption by about 86% while maintaining good accuracy in wavefield reconstruction. In addition, the issue of how to obtain a domain decomposition scheme with optimal performance is discussed in this paper.

Performance modeling and running strategy of parallel cdugksFOAM program

The cdugksFOAM program realizes the physical space grid and the velocity space grid in parallel at the same time. Its distinguishing feature lies in its potential for large-scale parallelism. However, the running time of the cdugksFOAM program is significantly dependent on the number of physical and velocity space partitions. In order to find the optimal partitioning strategies for a specific CFD problem running on a parallel computer, we performed performance modeling of the cdugksFOAM program. Firstly, we proposed a floating-point operations model, a MPI communication volume model, and a memory consumption model. Based on these models, we established a Roofline model to predict the computational time, and a model to predict communication time. According to the computational time model and the communication time model, the execution time model was proposed and its effectiveness was verified with two cases. Finally, the optimal running strategy that minimizes the product of the number of computing nodes and execution time was identified, providing meaningful guidance for the economic execution of the program.

Enhancing MPI remote memory access model for distributed-memory systems through one-sided broadcast implementation

Efficiently processing vast and expanding data volumes is a pressing challenge. Traditional high-performance computers, utilizing distributed-memory architecture and a message-passing model, grapple with synchronization issues, hampering their ability to keep up with the growing demands. Remote Memory Access (RMA), often referred to as one-sided MPI communications, offers a solution by allowing a process to directly access another process’s memory, eliminating the need for message exchange and significantly boosting performance. Unfortunately, the existing MPI RMA standard lacks a collective operation interface, limiting efficiency. To overcome this constraint, we introduce an algorithm design that enables efficient parallelizable collective operations within the RMA framework. Our study focuses primarily on the advantages of collective operations, using the broadcast algorithm as a case study. Our implementations surpass traditional methods, highlighting the promising potential of this technique, as indicated by initial performance tests.

A transmission optimization method for MPI communications

A transmission optimization method for MPI communications

LibFastMesh: An optimized finite-volume framework for computational aeroacoustics

This paper presents an optimized framework for simulating aeroacoustic flow on unstructured meshes. The libFastMesh (LFM) framework is based on a low-dissipation finite–volume discretization, together with high-order explicit time integration, for direct computation of aeroacoustic fields. These methods were previously implemented in caafoam, an OpenFOAM-based solver developed by D'Alessandro et al. [1] to resolve aeroacoustic flows. The new framework is specifically developed from scratch to alleviate two of the main bottlenecks currently limiting the performance of OpenFOAM-based solvers at large scale: memory bandwidth and inter-node communication. Compact data structures and compute kernel fusion are employed to enhance cache utilization and re-use, respectively. Separate treatment of inter-process boundary cells, together with non-blocking MPI communication enable effective overlap between communication and flux computations. The accuracy and robustness of libFastMesh are established via simulations of well-established aeroacoustic flow benchmarks. Comparison to previous simulation results obtained with caafoam, and to DNS data serve to validate the code. Finally, the effectiveness of the aforementioned optimizations is demonstrated through rigorous analysis of the proposed solver performance in single-node and multi-node operation modes. The obtained results show that libFastMesh offers a speed-up of up to 20x with respect to the previously developed OpenFOAM-based solver, caafoam, in large-scale computations. Moreover, LFM is shown to scale extremely for cell-per-core ratios of less than 500. These results are particularly appealing given the highly resource-demanding nature of direct computational aeroacoustics (CAA). Consequently, LFM could be an extremely attractive tool for scientists who wish to conduct large-scale CAA simulations on modern, exascale architectures. Program summaryProgram Title: libFastMeshCPC Library link to program files:https://doi.org/10.17632/7w9fy2xtcf.1Developer's repository link:https://github.com/TRC-HPC/LFM_PublicLicensing provisions: GPLv3Programming language: C++Nature of problem: This software solves compressible Navier–Stokes equations. The adopted numerics and flow models allow to capture with high accuracy aerodinamically generated sound. The solver is also designed to take advantage of the massively parallel computing systems which are strictly required for facing real–life aeroacoustic problems.Solution method: The libFastMesh (LFM) framework is based on a low-dissipation finite–volume discretization, together with high-order explicit time integration, for direct computation of aeroacoustic fields on unstructured meshes. Convective terms can be approximated using either the Kurganov–Noelle–Petrova (KNP) scheme or Pirozzoli's energy conserving approach. Time integration is fully–explicit and relies on a low-storage, 4th order Runge–Kutta scheme. The solver is specifically developed to run efficiently on modern many-core CPUs, and at large-scale where it exhibits excellent strong scaling features. This is made possible thanks to several optimizations that alleviate memory bandwidth and inter-node communication bottlenecks.Additional comments: OpenFOAM-v2112+ is an installation prerequisite. Standard OpenFOAM pre-processing and post-processing tools, as well as mesh decomposition and reordering methods may be used with the solver.

A multithreaded CUDA and OpenMP based power‐aware programming framework for multi‐node GPU systems

SummaryIn the article, we have proposed a framework that allows programming a parallel application for a multi‐node system, with one or more graphical processing units (GPUs) per node, using an OpenMP+extended CUDA API. OpenMP is used for launching threads responsible for management of particular GPUs and extended CUDA calls allow to transfer data and launch kernels on local and remote GPUs. The framework hides inter‐node MPI communication from the programmer. For optimization, the implementation takes advantage of the MPI_THREAD_MULTIPLE mode allowing: multiple threads handling distinct GPUs as well as overlapping communication and computations transparently using multiple CUDA streams. The solution allows data parallelization across available GPUs in order to minimize execution time and supports a power‐aware mode in which GPUs are automatically selected for computations using a greedy approach in order not to exceed an imposed power limit. We have implemented and benchmarked three parallel applications including: finding the largest divisors; verification of the Collatz conjecture; finding patterns in vectors. These were tested on three various systems: a GPU cluster with 16 nodes, each with NVIDIA GTX 1060 GPU; a powerful 2‐node system—one node with 8 NVIDIA Quadro RTX 6000 GPUs, the second with 4 NVIDIA Quadro RTX 5000 GPUs; a heterogeneous environment with one node with 2 NVIDIA RTX 2080 and 2 nodes with NVIDIA GTX 1060 GPUs. We demonstrated effectiveness of the framework through execution times versus power caps within ranges of 100–1400 W, 250–3000 W, and 125–600 W for these systems respectively as well as gains from using two versus one CUDA streams per GPU. Finally, we have shown that for the testbed applications the solution allows to obtain high speed‐ups between 89.3% and 97.4% of the theoretically assessed ideal ones, for 16 nodes and 2 CUDA streams, demonstrating very good parallel efficiency.

Parallel transient response topology optimization based on PETSc

Improving the dynamic properties and reducing the vibration levels is extremely important for the design of many engineering structures. In this paper, we develop a dynamic topology optimization program based on parallel computing with the help of the PETSc Portable Extension Toolkit for Scientific Computing and the MPI communication protocol. The SIMP model is used as the topology optimization model, and the sensitivity analysis schemes of the dynamic compliance and displacement response as opposed to the density design variables are given in the paper. Using this optimization formulation, we developed a parallel topology optimization program for minimizing the transient response. The convergence performance of the program is shown with a relatively large-scale numerical example.

COFFEE: Cross-Layer Optimization for Fast and Efficient Executions of Sinkhorn-Knopp Algorithm on HPC Systems

In this paper, we present COFFEE, cross-layer optimization for fast and efficient executions of the Sinkhorn-Knopp (SK) algorithm on HPC systems with clusters of compute nodes by exploring some architectural features of the system. By analyzing the performance of a typical implementation of the SK algorithm on such a system, a huge performance gap is observed between the row rescaling and column rescaling of the algorithm, where the latter requires much more time than the former. We also found that the costly MPI communication of the column rescaling seriously hinders the exploitation of parallelism. By observing and leveraging unique architectural characteristics across different system optimizations, such as column rescaling redesign, data blocking, micro-kernel design, enhanced intra-node and inter-node communication in MPI, etc., COFFEE is able to explore cross-layer optimization opportunities that enable fast and efficient execution of the SK algorithm. Our experimental results show that COFFEE provides up to 7.5X with an average of 2.0X performance improvement over the typical implementation on a single node, and up to 2.9X with an average of 1.6X performance improvement over the state-of-the-art MPI Allreduce algorithms on Tianhe-1 supercomputer.

Making applications faster by asynchronous execution: Slowing down processes or relaxing MPI collectives

Comprehending the performance bottlenecks at the core of the intricate hardware–software interactions exhibited by highly parallel programs on HPC clusters is crucial. This paper sheds light on the issue of automatically asynchronous MPI communication in memory-bound parallel programs on multicore clusters and how it can be facilitated. For instance, slowing down MPI processes by deliberate injection of delays can improve performance if certain conditions are met. This leads to the counter-intuitive conclusion that noise, independent of its source, is not always detrimental but can be leveraged for performance improvements. We employ phase-space graphs as a new tool to visualize parallel program dynamics. They are useful in spotting certain patterns in parallel execution that will easily go unnoticed with traditional tracing tools. We investigate five different microbenchmarks and applications on different supercomputer platforms: an MPI-augmented STREAM Triad, two implementations of Lattice-Boltzmann fluid solvers (D3Q19 and SPEChpc D2Q37), the LULESH and HPCG proxy applications.

HipBone: A performance-portable graphics processing unit-accelerated C++ version of the NekBone benchmark

We present hipBone, an open-source performance-portable proxy application for the Nek5000 (and NekRS) computational fluid dynamics applications. HipBone is a fully GPU-accelerated C++ implementation of the original NekBone CPU proxy application with several novel algorithmic and implementation improvements which optimize its performance on modern fine-grain parallel GPU accelerators. Our optimizations include a conversion to store the degrees of freedom of the problem in assembled form in order to reduce the amount of data moved during the main iteration and a portable implementation of the main Poisson operator kernel. We demonstrate near-roofline performance of the operator kernel on three different modern GPU accelerators from two different vendors. We present a novel algorithm for splitting the application of the Poisson operator on GPUs which aggressively hides MPI communication required for both halo exchange and assembly. Our implementation of nearest-neighbor MPI communication then leverages several different routing algorithms and GPU-Direct RDMA capabilities, when available, which improves scalability of the benchmark. We demonstrate the performance of hipBone on three different clusters housed at Oak Ridge National Laboratory, namely, the Summit supercomputer and the Frontier early-access clusters, Spock and Crusher. Our tests demonstrate both portability across different clusters and very good scaling efficiency, especially on large problems.

PaScaL_TDMA 2.0: A multi-GPU-based library for solving massive tridiagonal systems

PaScaL_TDMA 2.0: A multi-GPU-based library for solving massive tridiagonal systems

Detailed Modeling of Heterogeneous and Contention-Constrained Point-to-Point MPI Communication

The network topology of modern parallel computing systems is inherently heterogeneous, with a variety of latency and bandwidth values. Moreover, contention for the bandwidth can exist on different levels when many processes communicate with each other. Many-pair, point-to-point MP communication is thus characterized by heterogeneity and contention, even on a cluster of homogeneous multicore CPU nodes. To get a detailed understanding of the individual communication cost per MPI process, we propose a new modeling methodology that incorporates both heterogeneity and contention. First, we improve the standard max-rate model to better quantify the actually achievable bandwidth depending on the number of MPI processes in competition. Then, we make a further extension that more detailedly models the bandwidth contention when the competing MPI processes have different numbers of neighbors, with also non-uniform message sizes. Thereafter, we include more flexibility by considering interactions between intra-socket and inter-socket messaging. Through a series of experiments done on different processor architectures, we show that the new heterogeneous and contention-constrained performance models can adequately explain the individual communication cost associated with each MPI process. The largest test of realistic point-to-point MPI communication involves 8,192 processes and in total 2,744,632 simultaneous messages over 64 dual-socket AMD Epyc Rome compute nodes connected by InfiniBand, for which the overall prediction accuracy achieved is 8 4%.

An introspection monitoring library to improve MPI communication time

In this paper, we describe how to improve communication time of MPI parallel applications with the use of a library that enables to monitor MPI applications and allows for introspection (the program itself can query the state of the monitoring system). Based on previous work, this library is able to see how collective communications are decomposed into point-to-point messages. It also features monitoring sessions that allow suspending and restarting the monitoring, limiting it to specific portions of the code. Experiments show that the monitoring overhead is very small and that the proposed features allow for dynamic and efficient rank reordering enabling up to 2-time reduction of communication parts of some program.

A direct-hybrid CFD/CAA method based on lattice Boltzmann and acoustic perturbation equations

The accuracy of two direct coupled two-step CFD/CAA methods is discussed. For the flow field either a finite-volume (FV) method for the solution of the Navier–Stokes equations or a lattice Boltzmann (LB) method is coupled to a discontinuous Galerkin (DG) method for the solution of the acoustic perturbation equations. The coupling takes advantage of a joint Cartesian mesh allowing for the exchange of the acoustic sources without MPI communication. An immersed boundary treatment of the acoustic scattering from solid bodies by a novel solid wall formulation is implemented and validated in the DG method. Results for the case of a spinning vortex pair and the low Reynolds number unsteady flow around a circular cylinder show that a solution with comparable accuracy is obtained for the two direct-hybrid methods when using identical mesh resolution.

Near-Lossless MPI Tracing and Proxy Application Autogeneration

Traces of MPI communications are used by many performance analysis and visualization tools. Storing exhaustive traces of large-scale MPI applications is infeasible, however, because of their large volume. Aggregated or lossy MPI traces are smaller but provide much less information. In this paper we present Pilgrim, a near-lossless MPI tracing tool that, by using sophisticated compression techniques, generates small trace files at large scales and incurs only moderate overheads. We perform comprehensive studies of various compression techniques used for storing timestamps associated with each call. This timing information is essential for analysis purposes such as skews study. To demonstrate the usefulness of the detailed information stored by Pilgrim, we present a proxy application generator that can generate proxy apps that preserve original communication patterns from the Pilgrim traces.

GPU-accelerated DNS of compressible turbulent flows

This paper explores strategies to transform an existing CPU-based high-performance computational fluid dynamics solver, HyPar, for compressible flow simulations on emerging exascale heterogeneous (CPU+GPU) computing platforms. The scientific motivation for developing a GPU-enhanced version of HyPar is to simulate canonical turbulent flows at the highest resolution possible on such platforms. We show that optimizing memory operations and thread blocks results in 200x speedup of computationally intensive kernels compared with a CPU core. Using multiple GPUs and CUDA-aware MPI communication, we demonstrate both strong and weak scaling of our GPU-based HyPar implementation on the NVIDIA Volta V100 GPUs. We simulate the decay of homogeneous isotropic turbulence in a triply periodic box on grids with up to 10243 points (5.3 billion degrees of freedom) and on up to 1,024 GPUs. We compare the wall times for CPU-only and CPU+GPU simulations. The results presented in the paper are obtained on the Summit and Lassen supercomputers at Oak Ridge and Lawrence Livermore National Laboratories, respectively.

Evaluation of the Performance of Tightly Coupled Parallel Solvers and MPI Communications in IaaS From the Public Cloud

IaaS from the public cloud is becoming a new option for organizations in need of HPC capabilities. IaaS offers greater flexibility in hardware choices and operational costs. Furthermore, IaaS enables small organizations to access resources previously reserved for organizations with larger capitals. This article uses the HPCG benchmark to assess the performance of parallel solvers, which are critical in computational engineering, and microbenchmarks to measure collective MPI operations. The benchmarks encompass IaaS from five cloud vendors (AWS, Google Cloud Platform, Azure, Oracle CIoud Infrastructure, and Packet), and two architectures, ARM and x86_64. The benchmarks cover clusters with up to 4500 cores and illustrate the benefits of higher network bandwidth when scaling up clusters. The results for some of the clusters are particularly promising as they exhibit good scalability and compare well to on-premises supercomputers. Additionally, the study includes a preliminary cost estimate based on on-demand prices for IaaS computational power and memory.

GenASiSBasics: Object-oriented utilitarian functionality for large-scale physics simulations (Version 4)

GenASiSBasics: Object-oriented utilitarian functionality for large-scale physics simulations (Version 4)