MPI Library Research Articles

Taking the MPI standard and the open MPI library to exascale

The Open MPI for Exascale (OMPI-X) project was one of two in the Exascale Computing Project (ECP) focused on advancing the MPI ecosystem. The OMPI-X team worked with other MPI Forum members to champion several important features for inclusion in the MPI 4.0, 4.1, and upcoming 5.0 MPI standard versions, in support of the needs of exascale applications and systems. The team also worked with the larger Open MPI community to bring implementations of these new features and other enhancements into Open MPI, one of the leading open-source implementations of the MPI interface. This paper describes the motivation for the work of the OMPI-X project in the context of exascale computing needs, the nature of the resulting new capabilities in the MPI standard, and how they were implemented in the Open MPI library. Features include improved support for “MPI + X” programming models through partitioned communications and support for user-level threading, sessions, fault tolerance through the user-level fault mitigation (ULFM) and Reinit models, and other features. We also discuss enhancements to Open MPI providing improved performance and scalability for existing features, such as collective operations, one-sided operations, support for the Slingshot-11 interconnect of the initial exascale systems, and how the OMPI-X team worked to improve quality assurance for the Open MPI library, particularly on platforms of interest to the Department of Energy community.

Portability and scalability evaluation of large-scale statistical modeling and prediction software through HPC-ready containers

HPC-based applications often have complex workflows with many software dependencies that hinder their portability on contemporary HPC architectures. In addition, these applications often require extraordinary efforts to deploy and execute at performance potential on new HPC systems, while the users expert in these applications generally have less expertise in HPC and related technologies. This paper provides a dynamic solution that facilitates containerization for transferring HPC software onto diverse parallel systems. The study relies on the HPC Workflow as a Service (HPCWaaS) paradigm proposed by the EuroHPC eFlows4HPC project. It offers to deploy workflows through containers tailored for any of a number of specific HPC systems. Traditional container image creation tools rely on OS system packages compiled for generic architecture families (x86_64, amd64, ppc64, …) and specific MPI or GPU runtime library versions. The containerization solution proposed in this paper leverages HPC Builders such as Spack or Easybuild and multi-platform builders such as buildx to create a service for automating the creation of container images for the software specific to each hardware architecture, aiming to sustain the overall performance of the software. We assess the efficiency of our proposed solution for porting the geostatistics ExaGeoStat software on various parallel systems while preserving the computational performance. The results show that the performance of the generated images is comparable with the native execution of the software on the same architectures. On the distributed-memory system, the containerized version can scale up to 256 nodes without impacting performance.

Direct numerical simulation of three-dimensional particle-laden thermal convection using the Lattice Boltzmann Method

In thermal multiphase flows, the modulation in the heat transfer rate due to the presence of finite-size solid particles attract growing interest in recent years. This study focuses on developing a robust and efficient simulation method for particle-laden Rayleigh-Bénard convection. The present work utilizes the double distribution function-based thermal lattice-Boltzmann method (TLBM), which enables successful simulations of multiphase fluid-thermal interactions. The no-slip boundary of the moving solid particles is handled by the interpolated bounce-back scheme. Additionally, Galilean invariant momentum exchange and heat exchange approaches are employed for hydrodynamic force and heat transfer calculation at the solid boundaries. The accuracy of the current method is verified with several benchmark cases, including three-dimensional single-phase Rayleigh-Bénard convection, as well as the settling of hot and cold spherical particles in a three-dimensional enclosure. Furthermore, we explore the modulation of Rayleigh-Bénard flows due to the presence of freely moving finite-size particles. The simulations are conducted on a distributed memory cluster with a 3D domain decomposition technique facilitated by the MPI library. A brief discussion on the parallel performance of the simulations is provided for both particle-laden and single-phase flow scenarios. Finally, the effects of finite-size solid particles on the overall heat transfer efficiency and modulation to the flow field in three-dimensional particle-laden turbulent Rayleigh-Bénard convection are discussed. The results show that addition of solid particles results in a moderate increase in the overall Nusselt number, and this enhancement is mainly due to the increased heat flux transported by the particles.

MPI reduction and broadcast algorithms for computer clusters with multistage interconnection networks

Algorithms for MPI Allreduce, Bcast and Reduce collective operations have been developed. They are based on the dynamic construction of process groups at all levels of the memory hierarchy within the computing node and levels of the multistage interconnection network. The description of the network topology is loaded from an external file. Аlgorithms are implemented on the basis of the Open MPI library in an isolated collective component. Experiments on a cluster with a two-tier InfiniBand EDR network have shown the effectiveness of the algorithms for messages up to 16KB.

AliSim-HPC: parallel sequence simulator for phylogenetics.

Sequence simulation plays a vital role in phylogenetics with many applications, such as evaluating phylogenetic methods, testing hypotheses, and generating training data for machine-learning applications. We recently introduced a new simulator for multiple sequence alignments called AliSim, which outperformed existing tools. However, with the increasing demands of simulating large data sets, AliSim is still slow due to its sequential implementation; for example, to simulate millions of sequence alignments, AliSim took several days or weeks. Parallelization has been used for many phylogenetic inference methods but not yet for sequence simulation. This paper introduces AliSim-HPC, which, for the first time, employs high-performance computing for phylogenetic simulations. AliSim-HPC parallelizes the simulation process at both multi-core and multi-CPU levels using the OpenMP and message passing interface (MPI) libraries, respectively. AliSim-HPC is highly efficient and scalable, which reduces the runtime to simulate 100 large gap-free alignments (30 000 sequences of one million sites) from over one day to 11 min using 256 CPU cores from a cluster with six computing nodes, a 153-fold speedup. While the OpenMP version can only simulate gap-free alignments, the MPI version supports insertion-deletion models like the sequential AliSim. AliSim-HPC is open-source and available as part of the new IQ-TREE version v2.2.3 at https://github.com/iqtree/iqtree2/releases with a user manual at http://www.iqtree.org/doc/AliSim.

Massively Parallel Monte Carlo Sampling for Xinanjiang Hydrological Model Parameter Optimization Using CPU-GPU Computer Cluster

The Monte Carlo sampling (MCS) method is a simple and practical way for hydrological model parameter optimization. The MCS procedure is used to generate a large number of data points. Therefore, its computational efficiency is a key issue when applied to large-scale problems. The MCS method is an internally concurrent algorithm that can be parallelized. It has the potential to execute on massively parallel hardware systems such as multi-node computer clusters equipped with multiple CPUs and GPUs, which are known as heterogeneous hardware systems. To take advantage of this, we parallelize the algorithm and implement it on a multi-node computer cluster that hosts multiple INTEL multi-core CPUs and NVIDIA many-core GPUs by using C++ programming language combined with the MPI, OpenMP, and CUDA parallel programming libraries. The parallel parameter optimization method is coupled with the Xinanjiang hydrological model to test the acceleration efficiency when tackling real-world applications that have a very high computational burden. Numerical experiments indicate, on the one hand, that the computational efficiency of the massively parallel parameter optimization method is significantly improved compared to single-core CPU code, and the multi-GPU code achieves the fastest speed. On the other hand, the scalability property of the proposed method is also satisfactory. In addition, the correctness of the proposed method is also tested using sensitivity and uncertainty analysis of the model parameters. Study results indicate good acceleration efficiency and reliable correctness of the proposed parallel optimization methods, which demonstrates excellent prospects in practical applications.

A systematic review of the literature on methods and technologies for teaching parallel and distributed computing in universities

There is a growing demand for software developers who have experience writing parallel programs rather than just "parallelizing" sequential systems as computer hardware gets more and more parallel. In order to develop the skills of future software engineers, it is crucial to teach pupils parallelism in elementary computer science courses. We searched the Scopus database for articles on "teaching parallel and distributed computing" and "parallel programming," published in English between 2008 and 2019. 26 papers were included in the study after quality review. As a result, a lab course using the C++ programming language and MPI library serves as the primary teaching tool for parallel and distributed computing.

Parallelization of numerical solutions of shallow water equations by the finite volume method for implementation on multiprocessor systems and graphics processors

An overview of approaches to parallelization of grid-based numerical methods for solving shallow water equations for multiprocessor systems and graphics processors is presented. A multithreaded approach for shared-memory computing systems implemented on the basis of the OpenMP programming interface and a geometric decomposition approach with message-passing using the MPI library for distributed-memory computers are described. Multithreading for programming GPUs based on the OpenACC software interface is considered. For the COASTOX-UN system of two-dimensional modeling of hydrodynamics, sediment and radionuclide transport in river systems and coastal areas of the seas, the parallelization of its hydrodynamic model COASTOX-HD was carried out. In the developed numerical model, the shallow water equations are solved by finite-volume numerical methods on unstructured computational grids with triangular cells of variable size. The parallelization is implemented using a hybrid MPI+OpenACC approach targeting multiprocessor systems and GPUs. For multiprocessor computers, geometric decomposition and MPI-based messaging are used, and for GPUs, multithreading is implemented using OpenACC directives. The performance of the developed parallel hydrodynamic model was evaluated during the calculation of typical problems of hydrodynamics of shallow water bodies, river flood, and tsunami wave run-up on the coast on a Dell Precision Workstation 7920 multi-core workstation with two 20-core Intel Xeon Gold 6230 processors and NVIDIA Quadro RTX 5000 and NVIDIA GeForce RTX 3080 graphics cards. It is shown that the developed model has significantly accelerated the simulation on the considered multiprocessor system and the considered GPUs. The acceleration on GPUs depends on the size of the computational grid, increasing to saturation with an increase in the number of grid cells. It is established that for the developed parallel model, whose numerical schemes are related to algorithms with low computational intensity, the memory bandwidth of the NVIDIA architecture GPUs is a more important limiting factor of acceleration than their performance.

Research on Transient Characteristics of Scramjet Engine during Starting Sequence

Transient characteristics of entire engine system affect control method of the engine system. A quasi-transient simulator called Hypersonic Engine Dynamic Simulator (HEDS) was developed for the purpose of analyzing the transient behavior of a scramjet engine system. The simulator was constructed using the volume junction method, which is a type of finite volume method, and was introduced with a parallel computing function using the MPI library. In this study, the transient characteristics at the starting sequence of a scramjet engine equipped with an expander cycle was analyzed. In the startup transient analysis, two factors (the time from high-speed airflow inflow to the main valve opening and the time from the main valve opening to ignition) that affect the system startup were focused. In addition, constraints on the system were set to evaluate results of the analysis. As a result, there existed cases in which the maximum rotational speed of the turbo pump was much higher than expected when the interval between the inflow of high-speed airflow and the opening of the main valve was increased. In addition, there was also a delay time between ignition and fuel temperature rise. Therefore, the time from the main valve opening to ignition should be as short as possible. In the control phase, it was necessary to manipulate those two factors appropriately in order to bring the system to a steady state in a short time. HEDS was found to be a very effective tool for understanding the behavior of the system at the operating point and constructing a control method.

Shared Memory-based Algorithms for Vector Collective Operations of the MPI Standard

Algorithms for MPI_Scatterv, MPI_Gatherv, MPI_Allgatherv collective operations using the shared memory of multiprocessor SMP/NUMA systems are proposed. The algorithms use an approach based on copying message fragments via queues in a shared memory segment (Copy-In-Copy-Out). The algorithms were implemented based on Open MPI and the execution time is reduced on average by 20-40% for the MPI_Scatterv, MPI_Gatherv, MPI_Allgatherv operations compared to the coll/tuned component of the Open MPI library.

A methodology for assessing computation/communication overlap of MPI nonblocking collectives

A methodology for assessing computation/communication overlap of MPI nonblocking collectives

Heterogeneous Parallel Implementation of Large-Scale Numerical Simulation of Saint-Venant Equations

Large-scale floods are one of the major events that impact the national economy and people’s livelihood every year during the flood season. Predicting the factors of flood evolution is a worldwide problem. We use the two-dimensional Saint-Venant equations as an example and for high-performance computing in modelling the flood behavior. Discretization of the two-dimensional Saint-Venant equations with initial and boundary conditions with the finite difference method in the explicit leapfrog scheme is carried out. Afterwards, we employed a large-scale heterogeneous parallel solution on the “SunRising-1” supercomputer system using MPI, OpenMP, Pthread, and OpenCL runtime libraries. On this basis, we applied communication/calculation overlapping and the local memory acceleration to optimize the performance. Finally, various performance tests of the parallel scheme are carried out from different perspectives. We have found this method is efficient and recommend this approach be used in solving systems of partial differential equations similar to the Saint-Venant equations.

Using parallel computing to measure image processing on led walls on Albania’s national media

The use of AI technology has led to efficiency and better performance, as seen from the point of view of an economist, as efficiency may bring about, and thus better revenues. But the AI technology is also supplied by some important technologies such as parallel processing. This article aims to analyze the benefits of parallel processing by illustrating it with a code in C, using the MPI library. The experiment is done for the first time for academic purposes in a High-Performance Computer used in a university space, and then re-applied in the business space of the media which uses the latter kind of computer with some more advanced features to process the images in led walls. The results show that the image is the better and faster process when the number of nodes increases, but the way it behaves depends on the relation between the number of processes and the number of processors in the nodes. Keywords: Image processing; High-Performance Computing; Parallel Processing; media; MPI.

Analysis of shock wave-boundary layer interaction in a shock tube using higher order scheme

Over the last couple of decades, the shock-wave boundary-layer interaction has gained a lot of attention due to its practical importance in many engineering applications. It is a complex problem involving shock-wave bifurcation, boundary-layer separation, the interaction of contact discontinuity with the shock wave, the formation of shocklets, and the evolution of vortical structures having different length scales. It is difficult to capture experimentally the detailed flow field having the shocklets and vortices. Numerical solvers having negligible numerical dissipation are highly essential to predict these structures accurately. Over the years, many researchers have obtained a grid-converged solution for shock-wave boundary-layer interaction at Reynolds numbers of 200 and 1000. The shock-wave boundary-layer interaction at higher Reynolds numbers is not attempted due to the requirement of huge computational resources and challenges associated with convergence. In the investigation, the grid-converged solution is obtained for a Reynolds number of 2500 with a 13th order high-resolution hybrid scheme using 100 cores of a computational cluster equipped with 3.0 GHz Intel Xeon processors incorporating the MPI library for parallelization. The complex flow field is analysed in detail using wall density, density gradient, vorticity, pressure, and enstrophy plots after validating the solver with the benchmark wall pressure, density, and v-velocity around the primary vortex provided by Zhou et al. (2018), Phys. Fluids, 30, 016102 for a Reynolds number 1000. It is observed that the triple point height and number of vortices in both separated zone and at the shear layer increase with an increase in Reynolds number. The grid-converged data obtained from the present simulation can be used as benchmark data to validate different numerical schemes/solvers at higher Reynolds numbers.

High-performance pseudo-anonymization of virtual power plant data on a CPU cluster

The considerable move towards the use of renewable energy resources has been provided by the digitization of energy systems with the help of virtual power plants (VPPs). However, due to the coincidence of this move with the introduction of new technologies in information and communications, joining these systems raises concerns about the privacy of personal data. The only real-world approach widely used in this case is to anonymize or pseudonymize the information associated with individuals in data received from distributed measurement devices. In this paper, we propose the method of classifying received data packets into different flows and assigning different access levels for each flow. This method makes data pseudonymous. Before this step, the received data, which has a different format, is unionized. To implement this idea, a tuple space flow classification algorithm is parallelized on a CPU cluster using MPI and OpenMP according to different scenarios. The CPU cluster consists of one head node and two computational nodes for packet classification operations. In this research, two scenarios have been used to run the CPU algorithm in parallel. The first scenario uses MPI and the second scenario uses a combination of MPI and OpenMP libraries. Also, the Tuple Space algorithm has been implemented on the computing systems using the mentioned libraries in the form of two scenarios using OpenMP and MPI. According to our results, the increase in the number of processor cores is linearly correlated with the increase in the speed of classification. Furthermore, while MPI uses more memory than OpenMP, it helps to achieve a higher speed of classification. In the combined method, the maximum speed of flow classification can be achieved if the number of processes and threads is equal to the number of processor cores. In other words, when the sum of processes and threads does not outnumber CPU cores, the least classification time and memory usage can be achieved.

Simulation of two in-line wind turbines using an incompressible Finite Volume solver coupled with a Blade Element Model

The present study addresses the first steps of development and validation of a coupled Finite Volume-Blade Element Model (FV-BEM) simulation tool dedicated to offshore wind turbine farm modeling. The fluid domain is solved using the incompressible formulation of the CFD solver Grid-flow [40]. The turbines are taken into account using FAST (from NREL) [23] and their effects are imposed into the fluid domain through an actuator line model (ALM) for which specific enhancements are proposed. The Grid-flow solver and its two-way coupling with the aero-elastic modules from FAST are introduced and detailed. The present two-way coupling is built on a parallel framework using the MPI library. The ALM force coefficient values are deduced by FAST from the incident local velocity interpolated from the CFD solver grid at each timestep. In return, the ALM forces computed by FAST are projected from its cylindrical support to the Cartesian grid of the CFD solver. This force projection is made through a specific kernel interpolation technique inspired from meshless methods such as Smoothed Particle Hydrodynamics (SPH). The simulation of the case of two in-line wind turbines using the FV-BEM coupling proposed is presented, together with comparisons against experimental results for validation purposes. Coarse and refined grid results are compared and discussed. Finally, a discussion on performances, advantages and limitations of the formulation proposed is provided.

The relativistic Schrödinger equation through FFTW 3: An extension of quantumfdtd

In order to solve the time-independent three-dimensional Schrödinger equation, one can transform the time-dependent Schrödinger equation to imaginary time and use a parallelized iterative method to obtain the full three-dimensional eigen-states and eigen-values on very large lattices. In the case of the non-relativistic Schrödinger equation, there exists a publicly available code called quantumfdtd which implements this algorithm. In this paper, we (a) extend the quantumfdtd code to include the case of the relativistic Schrödinger equation and (b) add two optimized Fast Fourier Transform (FFT) based kinetic energy terms for non-relativistic cases. The new kinetic energy terms (two non-relativistic and one relativistic) are computed using the parallelized FFT-algorithm provided by the FFTW 3 library. The resulting quantumfdtd v3 code, which is publicly released with this paper, is backwards compatible with version 2, supporting explicit finite-differences schemes in addition to the new FFT-based schemes. Finally, we (c) extend the original code so that it supports arbitrary external file-based potentials and the option to project out distinct parity eigen-states from the solutions. Herein, we provide details of the quantumfdtd v3 implementation, comparisons and tests of the three new kinetic energy terms, and code documentation. Program summaryProgram Title:quantumfdtd v3CPC Library link to program files:https://doi.org/10.17632/9p8zyvmdy2.1Developer's repository link:https://github.com/quantumfdtd/quantumfdtd_v3Licensing provisions: GPLv3Programming language:C++, Python, ShellJournal reference of previous version: [1–4]Does the new version supersede the previous version?: YesReasons for the new version: We extended the previous version of quantumfdtd, which solves the time-independent non-relativistic three-dimensional Schrödinger equation, to the case of the relativistic kinetic energy. Additionally, we have added the capability of using arbitrary three-dimensional potentials from external files. This functionality is required, for instance, in order to interface with lattice QCD code.Summary of revisions: (a) Include the case of the relativistic Schrödinger equation, (b) add two optimized Fast Fourier Transform (FFT) based kinetic energy terms for non-relativistic cases, and (c) support arbitrary external file-based potentials and the option to project out distinct parity eigen-states from the solutions. The new kinetic energy terms (two non-relativistic and one relativistic) are computed using the parallelized FFT-algorithm provided by the FFTW 3 library.Nature of problem: We compute the ground, first, and second excited states of the time-independent three-dimensional Schrödinger equation. As input, we accept a number of hard-coded, analytical, three-dimensional potentials, as well as arbitrary potentials via external files. The outputs of the program are the corresponding eigen-values and eigen-vectors (energies and wave-functions). We can also project out distinct parity eigen-states of the solutions.Solution method: The time-dependent Schrödinger equation is transformed to imaginary time. We use a parallelized iterative method to obtain the full three-dimensional time-independent eigen-states and eigen-values on large lattices. The new kinetic energy terms (two non-relativistic and one relativistic) are computed using the parallelized FFT-algorithm provided by the FFTW 3 library. The old non-relativistic kinetic term is computed using the finite-difference time-domain (FDTD) algorithm. Finally, we provide several Python and Shell scripts for the analysis steps. This includes a Python program for projecting out distinct parity eigen-states from the solutions.Additional comments including restrictions and unusual features: We require the number of MPI processes being a divisor of the number N of spatial grid points. The following external programs/libraries are used and thus required:•MPI library•FFTW_MPI, version 3•GNU Scientific Library (GSL), linked to CBLAS•For some of the post-processing scripts, Python 3 is required References[1]A. Dumitru, Y. Guo, M. Strickland, Phys. Rev. D 79 (2009) 114003, https://doi.org/10.1103/PhysRevD.79.114003[2]A. Dumitru, Y. Guo, A. Mocsy, M. Strickland, Phys. Rev. D 79 (2009) 054019, https://doi.org/10.1103/PhysRevD.79.054019[3]M. Strickland, and D. Yager-Elorriaga, J. Compt. Phys. 229 (2010) 6015, https://doi.org/10.1016/j.jcp.2010.04.032[4]M. Margotta, K. McCarty, C. McGahan, M. Strickland, D. Yager-Elorriaga, Phys. Rev. D 83 (2011) 105019; Erratum: Phys. Rev. D 84 (2011) 069902, https://doi.org/10.1103/PhysRevD.83.105019

WannSymm: A symmetry analysis code for Wannier orbitals

We derived explicit expressions of symmetry operators on Wannier basis, and implemented these operators in WannSymm software. Based on this implementation, WannSymm can i) symmetrize the real-space Hamiltonian output from Wannier90 code, ii) generate symmetry operators of the little group at a specific k-point, and iii) perform symmetry analysis for Wannier band structure. In general, symmetrized Hamiltonians yield improved results compared with the original ones when they are employed for nodal structure searching, surface Green's function calculations, and other model calculations. Program summaryProgram Title: WannSymmCPC Library link to program files:https://doi.org/10.17632/zgf4p85pwx.1Licensing provisions: BSD 3-clauseProgramming language: CExternal libraries: spglib; BLAS and LAPACK; MPI libraries (optional)Nature of problem: Generate the symmetry operators in Wannier basis; calculate the symmetry eigenvalues and characters; symmetrize the real-space Hamiltonian according to the crystal structure. It deals with nonmagnetic or magnetic systems with or without spin-orbit coupling. For magnetic systems, it can deal with ferromagnetic, commensurate antiferromagnetic long range ordered collinear or non-collinear systems with spin-orbit coupling.Solution method: The code requires the real-space Hamiltonian from Wannier projection, the definition of Wannier basis and the crystal structure associated with the defined basis.

Principles of modeling the fluorescence spectral dynamics of dye molecules in solutions

FSDS is Fluorescence Spectral Dynamics Simulator, designed to investigate photoinduced charge transfer and its manifestations in time-resolved fluorescence spectra of a dye in a solvent. The project is implemented in C, using the MPI library. The computer model simulates stationary absorption and fluorescence spectra, as well as the time evolution of population distributions along the solvent reaction coordinate, to calculate spectral dynamics. Excitation and relaxation of intramolecular high-frequency vibrations are described at the quantum level. FSDS can fit the physical parameters of the dyes to experimental data. The dye excitation is consistently described accounting for the finite duration of the pump pulse. The program code implements two evolutionary models: a spin-boson model within the harmonic potential and a stochastic model based on the Smoluchowski diffusion operator for an arbitrary free energy surface. Computer simulation shows the numerical coincidence of the calculations performed in the framework of these models for the harmonic free energy surface. The detail description of the code and numerical schemes are presented. Program summaryProgram Title: Fluorescence Spectral Dynamics Simulator (FSDS)CPC Library link to program files:https://doi.org/10.17632/d39rrrkvhr.1Code Ocean capsule:https://codeocean.com/capsule/6544976Licensing provisions: GPLv3Programming language: CSupplementary material: Sample configuration files, User ManualNature of problem: The time-resolved spectra contain reach information about photochemical processes. However, the chemically important information about separate elementary processes is entangled due to they proceed in parallel. Time-resolved spectra modeling allows extracting information about these processes. The presented program code makes it possible to determine the relaxation characteristics of the solvent and the times of intramolecular relaxation.Solution method: Finite-difference time-domain and Brownian dynamics simulation methods.

Implementation and evaluation of MPI 4.0 partitioned communication libraries

Partitioned point-to-point communication primitives provide a performance-oriented mechanism to support a hybrid parallel programming model and have been included in the upcoming MPI-4.0 standard. These primitives enable an MPI library to transfer parts of the data buffer while the application provides partial contributions using multiple threads or tasks or simply pipelines the buffers sequentially. The focus of this paper is the design and implementation of a layered library that provides the functionality of these newer APIs and supports application development using these newer APIs. This library provides an opportunity to explore potential optimizations and identify further enhancements to the APIs. Initial experience in designing this library along with preliminary performance results are presented. In addition, the library is compared to initial prototype libraries that have recently become available that have been updated to the standard-compliant interface. These prototype libraries were built on remote-memory-access (RMA) primitives, offering insight into different implementation strategies. In general, we observe an interesting trade-off space, with the RMA-based implementation proving more performant for send-side partitioning, with increases in perceived bandwidth 8.9x on average over a single send, compared to the persistent-based implementation, which shows improvements 4.0x on average. In comparing the two implementations, we find that the persistent-based implementation enables more overlap for receive-side partitioning up to 5.37X the RMA library’s overlap, while the RMA-based implementation provides better send-side performance of up to 70%.