Message Passing Interface Library Research Articles

Accelerated numerical modeling of shallow water flows with MPI, OpenACC, and GPUs

In this paper, a time-explicit Finite-Volume method is adopted to solve the 2-D shallow water equations on an unstructured triangular mesh, using a two-stage Runge-Kutta integrator and a monotone MUSCL model to achieve second-order accuracy in time and space, respectively. A multi-GPU model is presented that uses the Message Passing Interface (MPI) with OpenACC and uses the METIS library to produce the domain decomposition. A CUDA-aware MPI library (GPUDirect) and overlapped MPI communication with computation are used to improve parallel performance. Two benchmark tests with wet and dry downstream beds are used to test the code's accuracy. Good results were achieved compared to the numerical simulations of published studies. Compared with the multi-CPU version of a 6-core CPU, maximum speedups of 56.18 and 331.51 were obtained using a single GPU and 8 GPUs, respectively. Higher mesh resolution enhances acceleration performance, and the model is applicable to other environmental modeling activities.

Wave–structure interaction by a two–way coupling between a fully nonlinear potential flow model and a Navier–Stokes solver

A two-way domain decomposition coupling procedure between a fully nonlinear potential flow model and a Navier–Stokes solver capturing the free surface with a Volume of Fluid method is used to study wave–structure interaction applied to offshore wind turbines. Away from the structure, the large-scale inviscid wave field is modeled by the potential code. Wave generation and absorption in this 3D hybrid model take place in the outer potential domain. The codes exchange data in the region around their common boundaries. Through the two-way coupling, waves propagate in and out of the viscous subdomain, making the hybrid algorithm suitable to study wave diffraction on marine structures, while keeping the viscous subdomain small. Each code uses its own mesh and time step. Subdomains are overlapping, therefore continuity conditions on velocity and free surface have to be verified on two distinct coupling surfaces at any time. Parallel implementation with communications between the models relying on the Message Passing Interface library allows calculations on large spatial and temporal scales. The coupling algorithm is first tested for regular nonlinear waves and then applied to simulate wave loads exerted on a vertical monopile in 3D. Attention is paid to the high-order components of the horizontal force.

MCMPI: A library with elasticity for multi‐domain and public cloud environments

SummaryThis article introduces a new library that leverages and extends the MPI (message passing interface) standard, capable of aggregating servers and clusters located in multiple domains, as well as cloud resources. The execution platform is created and presented to the MPI application transparently, without the need of recompiling any code. The solution also provides functions for provisioning, adding, and removing nodes at runtime, bringing elasticity to the application. Through benchmarks, its performance was compared with the native execution of the application using the native MPI library. A prototype of an elastic application was also developed, with positive results within the expected range.

A Parallel 3-D HIE-FDTD Method using the MPI Library

This paper presents the implementation of the parallel hybrid implicit-explicit finite-difference time-domain (HIE-FDTD) method using the Message Passing Interface (MPI) library. The method proves to be very effective in simulating large-scale three-dimensional electromagnetic problems with fine structures in one direction. For the decomposition of the computational volume in the HIE-FDTD method, an MPI Cartesian 2D topology is implemented, allowing arbitrary division of the volume in two directions. Derived data types provided in the MPI library are employed to optimize inter-process communication. High accuracy and efficiency are subsequently demonstrated through a numerical example of a frequency-selected surface (FSS). It shows that the proposed method is very suitable for parallel computing, and the parallel efficiency maintains above 80% for different numbers of processes.

SUARA: A scalable universal allreduce communication algorithm for acceleration of parallel deep learning applications

Parallel and distributed deep learning (PDNN) has become an effective strategy to reduce the long training times of large-scale deep neural networks. Mainstream PDNN software packages based on the message-passing interface (MPI) and employing synchronous stochastic gradient descent rely crucially on the performance of MPI allreduce collective communication routine.In this work, we propose a novel scalable universal allreduce meta-algorithm called SUARA. In general, SUARA consists of L serial steps, where L≥2, executed by all MPI processes involved in the allreduce operation. At each step, SUARA partitions this set of processes into subsets, which execute optimally selected library allreduce algorithms to solve sub-allreduce problems on these subsets in parallel, to accomplish the whole allreduce operation after completing all the L steps. We then design, theoretically study and implement a two-step SUARA (L=2) called SUARA2 on top of the Open MPI library. We prove that the theoretical asymptotic speedup of SUARA2 executed by P processes over the base Open MPI routine is O(P). Our experiments on Shaheen-II supercomputer employing 1024 nodes demonstrate over 2x speedup of SUARA2 over native Open MPI allreduce routine, which translates into the performance improvement of training of ResNet-50 DNN on ImageNet by 9%.

A massively parallel implicit 3D unstructured grid solver for computing turbulent flows on latest distributed memory computational architectures

An implicit unstructured grid density-based solver–PRAVAHA based on a parallel variant of the lower upper symmetric Gauss-Seidel (LUSGS) method is developed to compute large-scale engineering problems. A four-layered parallel algorithm is designed to efficiently compute three-dimensional turbulent flows on massively parallel modern multiple instruction multiple data-stream (MIMD) computational hardware. This data parallel approach achieves multiple layers of parallelism including continuity of flow solution, transfer of solution gradients, and calculation of drag/lift/solution residuals, right up to the innermost implicit LUSGS solver sub-routine, which is relatively less explored in the literature. Domain decomposition is performed using the METIS software based on multi-level graph partitioning algorithms. Non-blocking message-passing interface library functions are used to manage inter-processor communication through explicit message passing, efficiently. Super-linear scalability of the parallel solver is established on the current state-of-the-art supercomputing facility, the 838 teraflops PARAM seva on up to 6144 cores. Linear or even super-linear speedup on problems of significant size is observed even on ad-hoc parallel computing platforms like workstations and multi-node clusters, for turbulent flow simulations.

A numerical solver for investigating the space charge effect on theelectric field in liquid argon time projection chambers

This paper reports the development of a numerical solver aimed to simulate the interaction between the space charge (i.e. ions) distribution and the electric field in liquid argon time projection chamber (LArTPC) detectors. The ion transport equation is solved by a time-accurate, cell-centered finite volume method and the electric potential equation by a continuous finite element method. The electric potential equation updates the electric field which provides the drift velocity to the ion transport equation. The ion transport equation updates the space charge density distribution which appears as the source term in the electric potential equation. The interaction between the space charge distribution and the electric field is numerically simulated within each physical time step. The convective velocity in the ion transport equation can include the background flow velocity in addition to the electric drift velocity. The numerical solver has been parallelized using the Message Passing Interface (MPI) library. Numerical tests show and verify the capability and accuracy of the current numerical solver. It is planned that the developed numerical solver, together with a Computational Fluid Dynamics (CFD) package which provides the flow velocity field, can be used to investigate the space charge effect on the electric field in large-scale particle detectors.

Network-Assisted Noncontiguous Transfers for GPU-Aware MPI Libraries

The importance of graphics processing units (GPUs) in accelerating HPC applications is evident by the fact that a large number of supercomputing clusters are GPU enabled. Many of these HPC applications use message passing interface (MPI) as their programming model. These MPI applications frequently exchange data that is noncontiguous in GPU memory. MPI provides derived datatypes (DDTs) to represent such data. Past research on DDTs mainly focused on optimizing the pack–unpack kernels. Modern HCAs are capable of gathering/scattering data from/to noncontiguous GPU memory regions. We propose a low-overhead HCA-assisted scheme to improve the performance of GPU-based noncontiguous exchanges without the GPU-based pack–unpack kernels. We show that the proposed scheme provides up to 2× benefits compared to the existing pack-based scheme at the benchmark level. Furthermore, we show up to 17% improvement with the SW4Lite application compared to other MPI libraries, such as MVAPICH2-GDR and OpenMPI+UCX.

Design of a portable implementation of partitioned point‐to‐point communication primitives

AbstractThe Message Passing Interface (MPI) has been the dominant message passing solution for scientific computing for decades. MPI point‐to‐point communications are highly efficient mechanisms for process‐to‐process communication. However, MPI performance when processes utilize multiple threads is slowed by concurrency protections in the MPI library. MPI's current thread level interface imposes these overheads throughout the library when thread safety is needed. While much work has been done to reduce multithreading overheads in MPI, a solution is needed that reduces the number of messages exchanged in a threaded environment. Partitioned communication is included in the MPI 4.0 standard as an alternative that addresses the challenges of multithreaded communication in MPI today. Partitioned communication reduces overall message volume by creating a buffer‐sharing mechanism between threads such that they can indicate when portions of a communication buffer are available to be sent. Separation of the control and data planes in MPI is enabled by allowing persistent initialization and single occurrence message buffer matching from the indication that the data is ready to be sent. This enables the usage of underlying hardware primitives like triggered operations, where commands (destination, size, etc.) can be set up prior to data buffer readiness and readiness triggered with a simple doorbell/counter later. This approach is useful for future development of MPI operations in environments where traditional networking commands can have performance challenges, like accelerators (GPUs, FPGAs). In this paper, we detail the design and implementation of a layered library (built on top of MPI‐3.1) and an integrated Open MPI solution that supports the new, MPI‐4.0 partitioned communication feature set. The library will enable applications to use currently released MPI implementations and older legacy libraries to provide partitioned communication support while also enabling further exploration of this new communication model in new applications and use cases. We will compare the designs of the library and native Open MPI support, provide performance results and comparisons between the two approaches, and lessons learned from the implementation of partitioned communication in both library and native forms. We find that the native implementation and library have similar performance with a percentage difference under 0.94% in microbenchmarks and performance within 5% for a partitioned communication enabled proxy application.

Distributed Geometric Multigrid Method: Analysis of a V Cycle Truncation Level Criteria

This article presents the analysis of a V cycle truncation level criteria in a parallel implemention of a geometric multigrid method for solving partial differential equations, developed over a distributed memory system. The proposed system is implemented in C, using the Message Passing Interface library. A theoretical analysis of the proposed truncation level criteria is presented, and its evaluation is reported for the Poisson problem. The experimental analysis indicates that the proposed method achieves accurate speedup and computational efficiency, and shows a good scalability behavior to solve large problems by properly using more processing units.

Energy-efficient distributed password hash computation on heterogeneous embedded system

This paper presents the improved version of our cool Cracker cluster (cCc), a heterogeneous distributed system for parallel and energy-efficient bcrypt password hash computation. The cluster consists of up to 8 computational units (nodes) with different performances measured in bcrypt hash computations per second [H/s]. In the cluster, nodes are low-power heterogeneous embedded systems with programmable logic containing specialized hash computation accelerators. In the experiments, we used a combination of Xilinx Zynq-series SoC boards and ZTEX 1.15y board which was initially used as a bitcoin miner. Zynq based nodes use the improved version of our custom bcrypt accelerator, which executes the most costly parts of the bcrypt hash computation in programmable logic. The cluster was formed around the famous open-source password cracking software package John the Ripper (abbr. JtR). On the communication layer, we used Message Passing Interface (MPI)library with a standard Ethernet network connecting the nodes. To mitigate the different performances among the cluster nodes and to balance the load, we developed and implemented password candidate distribution scheme based on the passwords' probability distribution, i.e. the order of appearance in the dictionary. We tested individual nodes and the cluster as a whole, trying different combinations of nodes and evaluating our distribution scheme for password candidates. We also compared our cluster with various GPU implementations in terms of performance, energy-efficiency, and price-efficiency. We show that our solution outperforms other platforms such as high-end GPUs, by a factor of at least 3 in terms of energy-efficiency and thus producing less overall cost of password attack than other platforms. In terms of the total operational costs, our cluster pays off after 4500 cracked passwords for a bcrypt hash with cost parameter 12, which makes it more appealing for real-world password-based system attacks. We also demonstrate the scalability of our cCc cluster.

Analyzing the impact of the MPI allreduce in distributed training of convolutional neural networks

For many distributed applications, data communication poses an important bottleneck from the points of view of performance and energy consumption. As more cores are integrated per node, in general the global performance of the system increases yet eventually becomes limited by the interconnection network. This is the case for distributed data-parallel training of convolutional neural networks (CNNs), which usually proceeds on a cluster with a small to moderate number of nodes. In this paper, we analyze the performance of the Allreduce collective communication primitive, a key to the efficient data-parallel distributed training of CNNs. Our study targets the distinct realizations of this primitive in three high performance instances of Message Passing Interface (MPI), namely MPICH, OpenMPI, and IntelMPI, and employs a cluster equipped with state-of-the-art processor and network technologies. In addition, we apply the insights gained from the experimental analysis to the optimization of the TensorFlow framework when running on top of Horovod. Our study reveals that a careful selection of the most convenient MPI library and Allreduce (ARD) realization accelerates the training throughput by a factor of \(1.2\times \) compared with the default algorithm in the same MPI library, and up to \(2.8\times \) when comparing distinct MPI libraries in a number of relevant combinations of CNN model+dataset.

LWMPI: An MPI library for NoC‐based lightweight manycore processors with on‐chip memory constraints

AbstractLightweight manycore processors deliver high performance and energy efficiency by bundling hundreds of low‐power cores, a distributed memory architecture with small local memories and Networks‐on‐Chip in a single die. However, the lack of rich and portable programming models for these processors makes software development a challenging task. Currently, two approaches are employed to address programmability in lightweight manycores: Operating Systems (OSes) and baremetal runtime libraries. The former provides portability but exposes complex Operating System (OS)‐level programming interfaces to developers. The latter focuses on providing rich and high performance interfaces, which are vendor‐specific and yield to non‐portable software. In this work, we address these programmability and portability challenges by combining a rich OS with a well‐known standard for parallel programming. We propose a portable and lightweight Message Passing Interface (MPI) library (LWMPI) designed from scratch to cope with restrictions and intricacies of lightweight manycores. We integrated LWMPI into Nanvix, an open‐source distributed OS that runs on silicon lightweight manycores. The results obtained with a synthetic benchmark and a subset of the CAP Bench applications running on Kalray MPPA‐256 unveil that LWMPI not only delivers a lightweight and richer programming interface but also presents good performance and scalability results.

MPI collective communication through a single set of interfaces: A case for orthogonality

We present and discuss a unified view of and interface for collective communication in the MPI (Message-Passing Interface) standard that in a natural way exploits MPI’s orthogonality of concepts. We observe that the currently separate and different interfaces for sparse and global collective communication can be unified under the global collective communication interfaces, and at the same time lead to leaner and stronger support for stencil-like, sparse collective communication on Cartesian communicators. Our observations not only significantly reduce the number of concrete operation interfaces, but extend the functionality that can be supported by MPI while provisioning for possible future, much more wide-ranging functionality.We suggest to (re)define communicators as the sole carriers of the topological structure over processes that determines the semantics of the collective operations, and to limit the functions that can associate topological information with communicators to the functions for distributed graph topology and inter-communicator creation. As a consequence, one set of interfaces for collective communication operations (in blocking, non-blocking, and persistent variants) will suffice, thereby explicitly eliminating the MPI_Neighbor_ interfaces (in all variants) from the MPI standard. Topological structure will no longer be implied by Cartesian communicators, which in turn will have the sole role of naming processes in a (d-dimensional, Euclidean) geometric point-space. The geometric naming can be passed to the topology creating functions as part of the communicator, and guide process rank reordering and topological collective algorithm selection. We also explore ramifications of our proposal for one-sided communication.Concretely, at the price of only one essential, additional function, our suggestion eliminates 10 concrete collective function interfaces from MPI 3.1, and 15 from MPI 4.0, while providing vastly more optimization scope for the MPI library implementation. Interfaces for Cartesian communicators can likewise be simplified and/or eliminated from the MPI standard, as could the general active (post-start) synchronization mechanism for one-sided communication.

Implementation of the LAMMPS package using T-system with an Open Architecture

Supercomputer applications are usually implemented in the C, C++, and Fortran programming languages using different versions of the Message Passing Interface library. The "T-system" project (OpenTS) studies the issues of automatic dynamic parallelization of programs. In practical terms, the implementation of applications in a mixed (hybrid) style is relevant, when one part of the application is written in the paradigm of automatic dynamic parallelization of programs and does not use any primitives of the MPI library, and the other part of it is written using the Message Passing Interface library. In this case, the library is used, which is a part of the T-system and is called DMPI (Dynamic Message Passing Interface). In this way, it is necessary to evaluate the effectiveness of the MPI implementation available in the T-system. The purpose of this work is to examine the effectiveness of DMPI implementation in the T-system. In a classic MPI application, 0% of the code is implemented using automatic dynamic parallelization of programs and 100% of the code is implemented in the form of a regular Message Passing Interface program. For comparative analysis, at the beginning the code is executed on the standard Message Passing Interface, for which it was originally written, and then it is executed using the DMPI library taken from the developed T-system. Сomparing the effectiveness of the approaches, the performance losses and the prospects for using a hybrid programming style are evaluated. As a result of the conducted experimental studies for different types of computational problems, it was possible to make sure that the efficiency losses are negligible. This allowed to formulate the direction of further work on the T-system and the most promising options for building hybrid applications. Thus, this article presents the results of the comparative tests of LAMMPS application using OpenMPI and using OpenTS DMPI. The test results confirm the effectiveness of the DMPI implementation in the OpenTS parallel programming environment

An optimisation of allreduce communication in message-passing systems

Collective communication, namely the pattern allreduce in message-passing systems, is optimised based on measurements at the installation time of the library. The algorithms used are set up in an initialisation phase of the communication, as so-called persistent collective communication, introduced in the message-passing interface (MPI) standard. Part of our allreduce algorithms are the patterns reduce_scatter and allgatherv which are also considered standalone. For the allreduce pattern for short messages the existing cyclic shift algorithm (Bruck’s algorithm) is applied with a prefix operation. For allreduce and long messages our algorithm is based on reduce_scatter and allgatherv, where the cyclic shift algorithm is applied with a flexible number of communication ports per node. The algorithms for equal message sizes are used with non-equal message sizes together with a heuristic for rank reordering. Medium message sizes are communicated with an incomplete reduce_scatter followed by allgatherv. Furthermore, an optional recursive application of the cyclic shift algorithm is applied. All algorithms are applied at the node level. The data is gathered and scattered by the cores within the node and the communication algorithms are applied across the nodes. In general, our approach outperforms the non-persistent counterpart in established MPI libraries by up to one order of magnitude or shows equal performance, with a few exceptions of number of nodes and message sizes.

Hurricane Irma Simulation at South Florida Using the Parallel CEST Model

In this study, a parallel extension of the Coastal and Estuarine Storm Tide (CEST) model is developed and applied to simulate the storm surge tide at South Florida induced by hurricane Irma occurred in 2017. An improvement is also made to the existing advection algorithm in CEST. This is achieved through the introduction of high-order, monotone Semi-Lagrangian advection. Distributed memory parallelization is developed via the Message Passing Interface (MPI) library. The parallel CEST model can therefore be run efficiently on machines ranging from multicore laptops to massively High Performance Computing (HPC) system. The principle advantage of being able to run the CEST model on multiple cores is that relatively low run-time is possible for real world storm surge simulations on grids with high resolution, especially in the locality where the hurricane makes landfall. The computational time is critical for storm surge model forecast to finish simulations in 30 min, and results are available to users before the arrival of the next advisory. In this study, simulation of hurricane Irma induced storm surge was approximately 22 min for 4 day simulation, with the results validated by field measurements. Further efficiency analysis reveals that the parallel CEST model can achieve linear speedup when the number of processors is not very large.

An efficient parallel scheme based on the nodal discontinuous Galerkin method for fluid flow simulations

An efficient parallel scheme based on the nodal discontinuous Galerkin finite element method (nodal-DGFEM) for the numerical solution of the partial differential equations governing fluid flow phenomena is discussed. The flow solver is demonstrated to perform numerical simulation of two-dimensional flow regimes on unstructured triangular grids. The parallel implementation serves to fulfill the requisition of the numerical method regarding high-performance computing resources. The distributed memory programming model with the domain decomposition approach is adopted. The message passing interface library is used for communication among the parallel processes, which are assigned domain-decomposed subproblems. The presented parallelization strategy accurately and efficiently tackles the communication of multi-node data on the element edges between the neighboring parallel processes. The efficacy and efficiency of the parallel solver are demonstrated through solving the well-known problem of non-viscous isentropic convecting vortex flow on parallel systems. The parallelization would extend the scope of the DGFEM by producing solutions in reasonable time frames.

Speeding up the ODETTA code for solving particle transport problems

Mathematical simulation of fast neutron reactors requires high-precision calculations of protection problems based on unstructured meshes. The paper considers and analyzes a parallel version of the ODETTA code (Belousov et al. 2019) with the use of the MPI (Message Passing Interface) library technology (Knyazeva et al. 2006). The code is designed for numerical simulation of neutronic processes in shielding compositions of fast neutron lead cooled reactor plants in normal operating modes, and can be used to calculate the radiation conditions of using structural components and equipment of nuclear power facilities which are assumed to be the sources of and/or exposed to ionizing radiation during their safety justification. The operation of the generated code is compared against the previous version. The MPI-based development of the ODETTA code’s algorithmic part is described. Peculiarities and specific features of the code parallelization are presented, the code modification is given, and respective algorithms are considered. The structure of the ODETTA code based on the MPI is described in brief. The results of using the ODETTA code’s serial and parallel versions in OS Linux (Kostromin 2012) for NRNU MEPhI’s HPC cluster are provided (Savchenko et al. 2020). A comparative analysis is presented for two code implementation options in terms of speed and accuracy of results when using two different clusters and different numbers of nodes for these. Peculiarities of cluster-based calculations are noted.

Parallel Algorithm with Blocks for a Single-Machine Total Weighted Tardiness Scheduling Problem

In this paper, the weighted tardiness single-machine scheduling problem is considered. To solve it an approximate (tabu search) algorithm, which works by improving the current solution by searching the neighborhood, is used. Methods of eliminating bad solutions from the neighborhood (the so-called block elimination properties) were also presented and implemented in the algorithm. Blocks allow a significant shortening of the process of searching the neighborhood generated by insert type moves. The designed parallel tabu search algorithm was implemented using the MPI (Message Passing Interface) library. The obtained speedups are very large (over 60,000×) and superlinear. This may be a sign that the parallel algorithm is superior to the sequential one as the sequential algorithm is not able to effectively search the solution space for the problem under consideration. Only the introduction of diversification process through parallelization can provide an adequate coverage of the entire search process. The current methods of parallelization of metaheuristics give a speedup which strongly depends on the problem’s instances, rarely greater than number of used parallel processors. The method proposed here allows the obtaining of huge speedup values (over 60,000×), but only when so-called blocks are used. The above-mentioned speedup values can be obtained on high performance computing infrastructures such as clusters with the use of MPI library.