Open MPI Research Articles

Taking the MPI standard and the open MPI library to exascale

The Open MPI for Exascale (OMPI-X) project was one of two in the Exascale Computing Project (ECP) focused on advancing the MPI ecosystem. The OMPI-X team worked with other MPI Forum members to champion several important features for inclusion in the MPI 4.0, 4.1, and upcoming 5.0 MPI standard versions, in support of the needs of exascale applications and systems. The team also worked with the larger Open MPI community to bring implementations of these new features and other enhancements into Open MPI, one of the leading open-source implementations of the MPI interface. This paper describes the motivation for the work of the OMPI-X project in the context of exascale computing needs, the nature of the resulting new capabilities in the MPI standard, and how they were implemented in the Open MPI library. Features include improved support for “MPI + X” programming models through partitioned communications and support for user-level threading, sessions, fault tolerance through the user-level fault mitigation (ULFM) and Reinit models, and other features. We also discuss enhancements to Open MPI providing improved performance and scalability for existing features, such as collective operations, one-sided operations, support for the Slingshot-11 interconnect of the initial exascale systems, and how the OMPI-X team worked to improve quality assurance for the Open MPI library, particularly on platforms of interest to the Department of Energy community.

Bringing HPE Slingshot 11 support to Open MPI

Bringing HPE Slingshot 11 support to Open MPI

Prediction of mortality in secondary peritonitis: a prospective study comparing p-POSSUM, Mannheim Peritonitis Index, and Jabalpur Peritonitis Index

BackgroundRisk scoring systems are required to allow accurate prognostication, compare outcomes of surgery, and allow patients to make informed decisions about their health. This prospective study compares the p-POSSUM (Portsmouth Modification to Physiological and Operative Severity Score for Enumeration of Mortality), Mannheim Peritonitis Index, and Jabalpur Peritonitis Index for their utility in predicting mortality in patients with peritonitis.MethodsPerioperative data was collected from 235 patients with secondary peritonitis and used to calculate p-POSSUM, MPI, and JPI scores. The accuracy of the 3 scores was compared using receiver operator characteristic curves.Resultsp-POSSUM and Mannheim Peritonitis Index were similar in their accuracy with area under the curve (AUC) values of 0.756 and 0.757. Jabalpur Peritonitis Index had an AUC of 0.665.Conclusionp-POSSUM and Mannheim Peritonitis Index can be used to predict mortality in patients with secondary peritonitis. Jabalpur Peritonitis Index is not suited for this purpose. Further studies are required to improve the diagnostic performance of p-POSSUM and MPI in patients with secondary peritonitis.

MPI reduction and broadcast algorithms for computer clusters with multistage interconnection networks

Algorithms for MPI Allreduce, Bcast and Reduce collective operations have been developed. They are based on the dynamic construction of process groups at all levels of the memory hierarchy within the computing node and levels of the multistage interconnection network. The description of the network topology is loaded from an external file. Аlgorithms are implemented on the basis of the Open MPI library in an isolated collective component. Experiments on a cluster with a two-tier InfiniBand EDR network have shown the effectiveness of the algorithms for messages up to 16KB.

RAMP: A flat nanosecond optical network and MPI operations for distributed deep learning systems

Distributed deep learning (DDL) systems strongly depend on network performance. Current electronic packet switched (EPS) network architectures and technologies suffer from variable diameter topologies, low-bisection bandwidth and over-subscription affecting completion time of communication and collective operations. We introduce a near-exascale, full-bisection bandwidth, all-to-all, single-hop, all-optical network architecture with nanosecond reconfiguration called RAMP, which supports large-scale distributed and parallel computing systems (12.8 Tbps per node for up to 65,536 nodes). For the first time, a custom RAMP-x MPI strategy and a network transcoder is proposed to run MPI collective operations across the optical circuit switched (OCS) network in a schedule-less and contention-less manner. RAMP achieves 7.6-171× speed-up in completion time across all MPI operations compared to realistic EPS and OCS counterparts. It can also deliver a 1.3-16× and 7.8-58× reduction in Megatron and DLRM training time respectively while offering 38-47× and 6.4-26.5× improvement in energy consumption and cost respectively.

Multicenter matched-pair study comparing BACT/ALERT® MP reagent systems for the recovery of mycobacteria from specimens other than blood

The BACT/ALERT® MP Reagent System is a broth culture medium for optimal detection and recovery of mycobacteria from clinical samples. The MP formulation was recently modified to improve detection and recovery times. A multicenter prospective matched pair study design was conducted to validate the performance of improved MP (MP-I) versus current MP (MP-C) bottles utilizing nonsterile and normally sterile samples, except blood, from patients suspected of having mycobacterial infections. A total of 1488 clinical samples were collected to obtain 212 mycobacteria samples by either or both MP culture bottles. MP-I and MP-C sensitivities were 86.6% and 81.4%, respectively, but the difference was not significant (P = 0.163) while specificities were 96.8% and 93.8%, respectively, and that difference was significant (P = 0.002). Overall recovery was 94.34% for MP-I and 88.68% for MP-C (recovery was 100% for both bottles with 52 seeded samples). Overall performance of MP-I was better than MP-C for sensitivity, specificity, and recovery.

Can we predict death using scoring systems in patients with local peritonitis ? A retrospective study

Prognostic scores in patients with local peritonitis (LP) have not yet been studied exhaustively. We, therefore, aimed in this study to evaluate the ability of several scoring systems to predict death in LP. A retrospective analysis including 68 patients with LP was conducted at Prof. Dr. Stoyan Kirkovich University Hospital in Stara Zagora from January 2017 to August 2021. Clinical and laboratory data needed for calculating the scoring systems were collected at admission or postoperatively. We compared the prognostic performance of WSES SSS, MPI, SIRS, and qSOFA using area under the receiver operation characteristics (AUROC) curves and bivariate correlation analysis. The observed mortality rate was 8.8%. Among all scores, MPI showed the best prognostic performance (AUROC=0.805, 95%CI 0.660-0.950). A threshold MPI >25 points permitted prediction of adverse outcome with a sensitivity of 66.7% and a specificity of 80.6%. The only significant correlation was found between outcome and MPI (p=0.012, r=0.302).Conclusions: The MPI has the ability to prognosticate mortality in patients with LP unlike WSES SSS, qSOFA and SIRS.

Solving ARGESIM Benchmark CP2 with Open MPI and Matlab PCT – Lattice Boltzmann Simulation

Solving ARGESIM Benchmark CP2 with Open MPI and Matlab PCT – Lattice Boltzmann Simulation

Open structure magnetic particle imaging by nonlinear back projection tomography reconstruction.

In open structure MPI systems, the nonlinear variation of the field free lines in the large region of interest scanning process distorts the x-space image reconstruction. In this study, we propose a nonlinear field free line projection reconstruction algorithm to solve the edge distortion problem of open structure MPI imaging. First, we calculate the curvature change law of the field free line in the scanning process. Then, we design a nonlinear back projection reconstruction algorithm according to the nonlinear characteristics of the field free line in the scanning process. Finally, the nonlinear back projection reconstruction algorithm is used to complete the tomography of blood vessels. The numerical calculation and simulation results show that the open structure MPI combined with a nonlinear back projection reconstruction algorithm can accomplish vascular fault reconstruction. The reconstruction algorithm proposed in this paper suppresses the edge distortion of the image and improves the positioning accuracy of the image. The size of the region of interest where distortions are low is increased 16 times by allowing 10.9% degradation in the gradient. We provide a non-linear inverse projection reconstruction algorithm to reduce the structural artefacts caused by FFL distortion. It provides a reconstruction scheme for a large region of interest fine imaging of open structure FFL-MPI.

Shared Memory-based Algorithms for Vector Collective Operations of the MPI Standard

Algorithms for MPI_Scatterv, MPI_Gatherv, MPI_Allgatherv collective operations using the shared memory of multiprocessor SMP/NUMA systems are proposed. The algorithms use an approach based on copying message fragments via queues in a shared memory segment (Copy-In-Copy-Out). The algorithms were implemented based on Open MPI and the execution time is reduced on average by 20-40% for the MPI_Scatterv, MPI_Gatherv, MPI_Allgatherv operations compared to the coll/tuned component of the Open MPI library.

Evaluation of the Performance of Tightly Coupled Parallel Solvers and MPI Communications in IaaS From the Public Cloud

IaaS from the public cloud is becoming a new option for organizations in need of HPC capabilities. IaaS offers greater flexibility in hardware choices and operational costs. Furthermore, IaaS enables small organizations to access resources previously reserved for organizations with larger capitals. This article uses the HPCG benchmark to assess the performance of parallel solvers, which are critical in computational engineering, and microbenchmarks to measure collective MPI operations. The benchmarks encompass IaaS from five cloud vendors (AWS, Google Cloud Platform, Azure, Oracle CIoud Infrastructure, and Packet), and two architectures, ARM and x86_64. The benchmarks cover clusters with up to 4500 cores and illustrate the benefits of higher network bandwidth when scaling up clusters. The results for some of the clusters are particularly promising as they exhibit good scalability and compare well to on-premises supercomputers. Additionally, the study includes a preliminary cost estimate based on on-demand prices for IaaS computational power and memory.

Model-based selection of optimal MPI broadcast algorithms for multi-core clusters

The performance of collective communication operations determines the overall performance of MPI applications. Different algorithms have been developed and implemented for each MPI collective operation, but none proved superior in all situations. Therefore, MPI implementations have to solve the problem of selecting the optimal algorithm for the collective operation depending on the platform, the number of processes involved, the message size(s), etc. The current solution method is purely empirical.Recently, an alternative solution method using analytical performance models of collective algorithms has been proposed and proved both accurate and efficient for one-process-per-CPU configurations. The method derives the analytical performance models of algorithms from their code implementation rather than from high-level mathematical definitions, and estimates the parameters of the models separately for each algorithm. The method is network and topology oblivious and uses the Hockney model for point-to-point communications. In this paper, we extend that selection method to the case of clusters of multi-core processors, where each core of the platform runs a process of the MPI application.We present the proposed approach using Open MPI broadcast algorithms, and experimentally validate it on three different clusters of multi-core processors, Grisou, Gros and MareNostrum4.

Numerical Study on the Performance and NOx Emission Characteristics of an 800cc MPI Turbocharged SI Engine

The main purpose of this study is to optimize engine performance and emission characteristics of off-road engines with retarded spark timing compared to MBT by repurposing the existing passenger engine. This study uses a one-dimensional (1D)-simulation to develop a non-road gasoline MPI turbo engine. The SI turbulent flame model of the GT-suite, an operational performance predictable program, presents turbocharger matching and optimal operation design points. To optimize the engine performance, the SI turbulent model uses three operation parameters: spark timing, intake valve overlap, and boost pressure. Spark timing determines the initial state of combustion and thermal efficiency, and is the main variable of the engine. The maximum brake torque (MBT) point can be identified for spark timing, and abnormal combustion phenomena, such as knocking, can be identified. Spark timing is related to engine performance, and emissions of exhaust pollutants are predictable. If the spark timing is set to variables, the engine performance and emissions can be confirmed and predicted. The intake valve overlap can predict the performance and exhaust gas by controlling the airflow and combustion chamber flow, and can control the performance of the engine by controlling the flow in the cylinder. In addition, a criterion can be set to consider the optimum operating point of the non-road vehicle while investigating the performance and exhaust gas emissions accompanying changes in boost pressure With these parameters, the design of experiment (DoE) of the 1D-simulation is performed, and the driving performance and knocking phenomenon for each RPM are predicted during the wide open throttle (WOT) of the gasoline MPI Turbo SI engine. The multi-objective Pareto technique is also used to optimize engine performance and exhaust gas emissions, and to present optimized design points for the target engine, the downsized gasoline MPI Turbo SI engine. The results of the Pareto optimal solution showed a maximum torque increase of 12.78% and a NOx decrease of 54.31%.

Callback-based completion notification using MPI Continuations

Asynchronous programming models (APM) are gaining more and more traction, allowing applications to expose the available concurrency to a runtime system tasked with coordinating the execution. While MPI has long provided support for multi-threaded communication and non-blocking operations, it falls short of adequately supporting APMs as correctly and efficiently handling MPI communication in different models is still a challenge. We have previously proposed an extension to the MPI standard providing operation completion notifications using callbacks, so-called MPI Continuations. This interface is flexible enough to accommodate a wide range of different APMs.In this paper, we present an extension to the previously described interface that allows for finer control of the behavior of the MPI Continuations interface. We then present some of our first experiences in using the interface in the context of different applications, including the NAS parallel benchmarks, the PaRSEC task-based runtime system, and a load-balancing scheme within an adaptive mesh refinement solver called ExaHyPE. We show that the interface, implemented inside Open MPI, enables low-latency, high-throughput completion notifications that outperform solutions implemented in the application space.

Empirical Performance Evaluation of Communication Libraries for Multi-GPU based Distributed Deep Learning in a Container Environment

Recently, most cloud services use Docker container environment to provide their services. However, there are no researches to evaluate the performance of communication libraries for multi-GPU based distributed deep learning in a Docker container environment. In this paper, we propose an efficient communication architecture for multi-GPU based deep learning in a Docker container environment by evaluating the performances of various communication libraries. We compare the performances of the parameter server architecture and the All-reduce architecture, which are typical distributed deep learning architectures. Further, we analyze the performances of two separate multi-GPU resource allocation policies ― allocating a single GPU to each Docker container and allocating multiple GPUs to each Docker container. We also experiment with the scalability of collective communication by increasing the number of GPUs from one to four. Through experiments, we compare OpenMPI and MPICH, which are representative open source MPI libraries, and NCCL, which is NVIDIA’s collective communication library for the multi-GPU setting. In the parameter server architecture, we show that using CUDA-aware OpenMPI with multi-GPU per Docker container environment reduces communication latency by up to 75%. Also, we show that using NCCL in All-reduce architecture reduces communication latency by up to 93% compared to other libraries.

Designing a parallel Feel-the-Way clustering algorithm on HPC systems

This paper introduces a new parallel clustering algorithm, named Feel-the-Way clustering algorithm, that provides better or equivalent convergence rate than the traditional clustering methods by optimizing the synchronization and communication costs. Our algorithm design centers on how to optimize three factors simultaneously: reduced synchronizations, improved convergence rate, and retained same or comparable optimization cost. To compare the optimization cost, we use the Sum of Square Error (SSE) cost as the metric, which is the sum of the square distance between each data point and its assigned clusters. Compared with the traditional MPI k-means algorithm, the new Feel-the-Way algorithm requires less communications among participating processes. As for the convergence rate, the new algorithm requires fewer number of iterations to converge. As for the optimization cost, it obtains the SSE costs that are close to the k-means algorithm. In the paper, we first design the full-step Feel-the-Way k-means clustering algorithm that can significantly reduce the number of iterations that are required by the original k-means clustering method. Next, we improve the performance of the full-step algorithm by adopting an optimized sampling-based approach, named reassignment-history-aware sampling. Our experimental results show that the optimized sampling-based Feel-the-Way method is significantly faster than the widely used k-means clustering method, and can provide comparable optimization costs. More extensive experiments with several synthetic datasets and real-world datasets (e.g., MNIST, CIFAR-10, ENRON, and PLACES-2) show that the new parallel algorithm can outperform the open source MPI k-means library by up to 110% on a high-performance computing system using 4,096 CPU cores. In addition, the new algorithm can take up to 51% fewer iterations to converge than the k-means clustering algorithm.

Process arrival pattern aware algorithms for acceleration of scatter and gather operations

Imbalanced process arrival patterns (PAPs) are ubiquitous in many parallel and distributed systems, especially in HPC ones. The collective operations, e.g. in MPI, are designed for equal process arrival times, and are not optimized for deviations in their appearance. We propose eight new PAP-aware algorithms for the scatter and gather operations. They are binomial or linear tree adaptations introducing additional process ordering and (in some cases) additional activities in a special background thread. The solution was implemented using one of the most popular open source MPI compliant library (OpenMPI), and evaluated in a typical HPC environment using a specially developed benchmark as well as a real application: FFT. The experimental results show a significant advantage of the proposed approach over the default OpenMPI implementation, showing good scalability and high performance with the FFT acceleration for the communication run time: 16.7% and for the total application execution time: 3.3%.

Qprop with faster calculation of photoelectron spectra

The calculation of accurate photoelectron spectra (PES) for strong-field laser-atom experiments is a demanding computational task, even in single-active-electron approximation. The Qprop code, published in 2006, has been extended in 2016 in order to provide the possibility to calculate PES using the so-called t-SURFF approach [Tao and Scrinzi (2012)]. In t-SURFF, the flux through a surface while the laser is on is monitored. Calculating PES from this flux through a surface enclosing a relatively small computational grid is much more efficient than calculating it from the widely spread wavefunction at the end of the laser pulse on a much larger grid. However, the smaller the minimum photoelectron energy of interest is, the more post-propagation after the actual laser pulse is necessary. This drawback of t-SURFF has been overcome by Morales et al. [Morales et al. (2016)] by noticing that the propagation of the wavefunction from the end of the laser pulse to infinity can be performed very efficiently in a single step. In this work, we introduce Qprop 3.0, in which this single-step post-propagation (dubbed i-SURFV) is added. Examples, illustrating the new feature, are discussed. A few other improvements, concerning mainly the parameter files, are also explained. NEW VERSION PROGRAM SUMMARYProgram Title: QpropProgram Files doi:http://dx.doi.org/10.17632/cxj2ygn4ph.2Licensing provisions: GNU General Public License, version 3Programming language: C++External routines/libraries: GNU Scientific Library, Open MPI (optional).Journal reference of previous version: Comput. Phys. Comm. 207(2016) 452–463Does the new version supersede the previous version?: Fully supports the functionality of Qprop 2.0.Nature of problem: Efficient calculation of PES for typical strong-field and attosecond physics ionization scenarios.Solution method: The time-dependent Schrödinger equation is solved by propagating the electronic wavefunction using a Crank–Nicolson propagator. The wavefunction is represented by an expansion in spherical harmonics. The t-SURFF method in combination with i-SURFV is used to calculate PES.Reasons for the new version: The i-SURFV method is employed to speed up the calculation of PES.Summary of revisions: The i-SURFV method is implemented. A set of examples is provided.Additional comments including restrictions and unusual features: The atomic potential needs to be of finite range in case of t-SURFF/i-SURFV usage (i.e., the Coulomb tail is truncated at sufficiently large distances). The laser–matter interaction is described in dipole approximation and velocity gauge.For additional information see www.qprop.de

Using relaxed concurrent data structures for contention minimization in multithreaded MPI programs

Parallel computing is one of the top priorities in computer science. The main means of parallel processing information is a distributed computing system (CS) - a composition of elementary machines that interact through a communication medium. Modern distributed VSs implement thread-level parallelism (TLP) within a single computing node (multi-core CS with shared memory), as well as process-level parallelism (PLP) process-level parallelism for the entire distributed CS. The main tool for developing parallel programs for such systems is the MPI standard. The need to create scalable parallel programs that effectively use compute nodes with shared memory has determined the development of the MPI standard, which today supports the creation of hybrid multi-threaded MPI programs. A hybrid multi-threaded MPI program is the combination of the computational capabilities of processes and threads. The standard defines four types of multithreading: Single - one thread of execution; Funneled - a multi-threaded program, but only main thread can perform MPI operations; Serialized - only one thread at the exact same time can make a call to MPI functions; Multiple - each program flow can perform MPI functions at any time. The main task of the multiple mode is the need to synchronize the communication flows within each process. This paper presents an overview of the work that addresses the problem of synchronizing processes running on remote machines and synchronizing internal program threads. Method for synchronization of threads based on queues with weakened semantics of operations is proposed.

A computational framework with an adaptive mesh refinement technique for concentrated suspension flows

A computational framework (solver) is developed for incompressible isothermal concentrated non-colloidal suspension flows of rigid particles. The number of particles can be of the order of millions. The developed solver, named “SuspensionFoam” can predict the phenomenon of shear-induced migration in complex geometries in a continuum mechanics modeling approach. It employs the C++ object-oriented OpenFOAM library features including adaptive mesh refinement (AMR), dynamic mesh rotation in both 2D and 3D, adaptive time steps, multi-CPU parallel-run over MPI (and open MPI) and run-time selection of different liquid and particles properties (i.e. diameter, maximum packing). Three different numerical examples have been conducted by simulating isodense suspension flow in a vertical Couette cell, non-isodense resuspension and mixing of particles in a horizontal Couette cell, and isodense suspension flow in a conduit. The present AMR technique is found to reduce the computational process considerably, with a good accuracy. Numerical results are presented and discussed, with comparisons to experimental data.