Message Passing Interface Implementation Research Articles

Parallel computation to bidimensional heat equation using MPI/CUDA and FFTW package

In this study, we present a fast algorithm for the numerical solution of the heat equation. The heat equation models the heat diffusion over time and through a given region. We engage a finite difference method to solve this equation numerically. The performance of its parallel implementation is considered using Message Passing Interface (MPI), Compute Unified Device Architecture (CUDA), and time schemes, such as Forward Euler (FE) and Runge-Kutta (RK) methods. The originality of this study is research on parallel implementations of the fourth-order Runge-Kutta method (RK4) for sparse matrices on Graphics Processing Unit (GPU) architecture. The supreme proprietary framework for GPU computing is CUDA, provided by NVIDIA. We will show three metrics through this parallelization to compare the computing performance: time-to-solution, speed-up, and performance. The spectral method is investigated by utilizing the FFTW software library, based on the computation of the fast Fourier transforms (FFT) in parallel and distributed memory architectures. Our CUDA-based FFT, named CUFFT, is performed in platforms, which is a highly optimized FFTW implementation. We will give numerical tests to reveal that this method is up-and-coming for solving the heat equation. The final result demonstrates that CUDA has a significant advantage and performance since the computational cost is tiny compared with the MPI implementation. This vital performance gain is also achieved through careful attention of managing memory communication and access.

Design of a portable implementation of partitioned point‐to‐point communication primitives

AbstractThe Message Passing Interface (MPI) has been the dominant message passing solution for scientific computing for decades. MPI point‐to‐point communications are highly efficient mechanisms for process‐to‐process communication. However, MPI performance when processes utilize multiple threads is slowed by concurrency protections in the MPI library. MPI's current thread level interface imposes these overheads throughout the library when thread safety is needed. While much work has been done to reduce multithreading overheads in MPI, a solution is needed that reduces the number of messages exchanged in a threaded environment. Partitioned communication is included in the MPI 4.0 standard as an alternative that addresses the challenges of multithreaded communication in MPI today. Partitioned communication reduces overall message volume by creating a buffer‐sharing mechanism between threads such that they can indicate when portions of a communication buffer are available to be sent. Separation of the control and data planes in MPI is enabled by allowing persistent initialization and single occurrence message buffer matching from the indication that the data is ready to be sent. This enables the usage of underlying hardware primitives like triggered operations, where commands (destination, size, etc.) can be set up prior to data buffer readiness and readiness triggered with a simple doorbell/counter later. This approach is useful for future development of MPI operations in environments where traditional networking commands can have performance challenges, like accelerators (GPUs, FPGAs). In this paper, we detail the design and implementation of a layered library (built on top of MPI‐3.1) and an integrated Open MPI solution that supports the new, MPI‐4.0 partitioned communication feature set. The library will enable applications to use currently released MPI implementations and older legacy libraries to provide partitioned communication support while also enabling further exploration of this new communication model in new applications and use cases. We will compare the designs of the library and native Open MPI support, provide performance results and comparisons between the two approaches, and lessons learned from the implementation of partitioned communication in both library and native forms. We find that the native implementation and library have similar performance with a percentage difference under 0.94% in microbenchmarks and performance within 5% for a partitioned communication enabled proxy application.

Consensus-based Dantzig-Wolfe decomposition

Dantzig-Wolfe decomposition (DWD) is a classical algorithm for solving large-scale linear programs whose constraint matrix involves a set of independent blocks coupled with a set of linking rows. The algorithm decomposes such a model into a master problem and a set of independent subproblems that can be solved in a distributed manner. In a typical implementation, the master problem is solved centrally. In certain settings, solving the master problem centrally is undesirable or infeasible, such as in the case of decentralized storage of data, or when independent agents who are responsible for the subproblems desire privacy of information. In this paper, we propose a fully distributed DWD algorithm which relies on solving the master problem using a consensus-based Alternating Direction Method of Multipliers (ADMM) method. We derive error bounds on the optimality gap and feasibility violation of the proposed approach. We provide preliminary computational results for our algorithm using a Message Passing Interface implementation on a delivery planning problem, the multi-commodity network flow problem, and synthetic instances where we obtain high quality solutions. An open-source implementation of the algorithm is available.

Accelerating Deep Learning Using Interconnect-Aware UCX Communication for MPI Collectives

Deep learning workloads on modern multi-graphics processing unit (GPU) nodes are highly dependent on intranode interconnects, such as NVLink and PCIe, for high-performance communication. In this article, we take on the challenge to design an interconnect-aware multipath GPU-to-GPU communication using unified communication X (UCX) to utilize all available bandwidth for both NVLink-based systems and those that use a mixture of NVLink and PCIe. Our proposed multipath data transfer mechanism pipelines and stripes the message across multiple intrasocket communication channels and memory regions to achieve 1.84× higher bandwidth for Open message passing interface (MPI) on NVLink-based systems and 1.23× on NVLink and PCIe systems. We then utilize this mechanism to propose a three-stage hierarchical, pipelined MPI_Allreduce design as well as a flat pipelined two-stage algorithm for two different node topologies. For large messages, our proposed algorithms achieve a high speedup when compared to other MPI implementations. We also observe significant speedup for the proposed MPI_Allreduce with Horovod + TensorFlow with a variety of deep learning models.

MPI Associated Scalability of Open-Source CFD Codes for Oil Spill Assessment

General-purpose CFD codes have recently become an increasingly discussed alternative to standardized, simplified and usually empirically calibrated specialized tools for pollution analyses. Commonly, CFD codes tend to provide physically more sensible results and can indicate the underlying cause for a given problem. Use for ecological problems, however, has usually been avoided due to the sizes of computational domains and inherent complexity of the calculations that need to be conducted. Adoption in recent years is mostly driven by significant improvements in computational capabilities and advancements related to code and communication optimizations. Unfortunately, due to substantial branching of codes and accompanying indispensable communication routines, especially in open-source community, performance and consequently applicability of codes, can vary significantly. This article aims to outline key limitations and quantify performance gains which can be obtained in a high-performance computing environment through the use of different communication protocols, when evaluating typical pollution problems such as oil spills. Obtained results indicate that savings of up to 40% in computational time can be achieved, depending on the code and message passing interface implementation for a problem in question, thus demonstrating the importance of communication protocols.

DL_FFLUX: A Parallel, Quantum Chemical Topology Force Field.

DL_FFLUX is a force field based on quantum chemical topology that can perform molecular dynamics for flexible molecules endowed with polarizable atomic multipole moments (up to hexadecapole). Using the machine learning method kriging (aka Gaussian process regression), DL_FFLUX has access to atomic properties (energy, charge, dipole moment, etc.) with quantum mechanical accuracy. Newly optimized and parallelized using domain decomposition Message Passing Interface (MPI), DL_FFLUX is now able to deliver this rigorous methodology at scale while still in reasonable time frames. DL_FFLUX is delivered as an add-on to the widely distributed molecular dynamics code DL_POLY 4.08. For the systems studied here (103–105 atoms), DL_FFLUX is shown to add minimal computational cost to the standard DL_POLY package. In fact, the optimization of the electrostatics in DL_FFLUX means that, when high-rank multipole moments are enabled, DL_FFLUX is up to 1.25× faster than standard DL_POLY. The parallel DL_FFLUX preserves the quality of the scaling of MPI implementation in standard DL_POLY. For the first time, it is feasible to use the full capability of DL_FFLUX to study systems that are large enough to be of real-world interest. For example, a fully flexible, high-rank polarized (up to and including quadrupole moments) 1 ns simulation of a system of 10 125 atoms (3375 water molecules) takes 30 h (wall time) on 18 cores.

A novel MPI+MPI hybrid approach combining MPI-3 shared memory windows and C11/C++11 memory model

The increase of the number of cores in processors used in modern cluster architectures advocates hybrid parallel programming, combining Message Passing Interface (MPI) for internode operations and a shared memory treatment of intranode operations. We propose an MPI+MPI hybrid approach to parallel programming in which shared memory operations are managed by the combination of MPI shared memory windows introduced with MPI-3, C11/C++11 atomic operations and the associated multi-thread memory model. We illustrate the method on fundamental parallel operations (barrier, reduction) and on the ghost update, which is prevalent in many parallel numerical methods. The performance tests on Reedbush-U and Oakbridge-CX systems show that using the C11/C++11 memory model to manage shared memory windows can achieve levels of performance comparable to state of the art MPI implementations, while reducing the variance of execution times as well as increasing the level of synchronization between processes, especially in multiple nodes environments. It also reduces significantly the execution time of ghost updates compared to flat MPI, and the synchronization of shared data with the C++11 memory model is observed to be more efficient than other synchronization methods based on RMA utilities.

Mpi4py: Status Update After 12 Years of Development

MPI for Python (mpi4py) has evolved to become the most used Python binding for the message passing interface (MPI). We report on various improvements and features that mpi4py gradually accumulated over the past decade, including support up to the MPI-3.1 specification, support for CUDA-aware MPI implementations, and other utilities at the intersection of MPI-based parallel distributed computing and Python application development.

Exploring parallel MPI fault tolerance mechanisms for phylogenetic inference with RAxML-NG.

MotivationPhylogenetic trees are now routinely inferred on large scale high performance computing systems with thousands of cores as the parallel scalability of phylogenetic inference tools has improved over the past years to cope with the molecular data avalanche. Thus, the parallel fault tolerance of phylogenetic inference tools has become a relevant challenge. To this end, we explore parallel fault tolerance mechanisms and algorithms, the software modifications required and the performance penalties induced via enabling parallel fault tolerance by example of RAxML-NG, the successor of the widely used RAxML tool for maximum likelihood-based phylogenetic tree inference.ResultsWe find that the slowdown induced by the necessary additional recovery mechanisms in RAxML-NG is on average 1.00 ± 0.04. The overall slowdown by using these recovery mechanisms in conjunction with a fault-tolerant Message Passing Interface implementation amounts to on average 1.7 ± 0.6 for large empirical datasets. Via failure simulations, we show that RAxML-NG can successfully recover from multiple simultaneous failures, subsequent failures, failures during recovery and failures during checkpointing. Recoveries are automatic and transparent to the user.Availability and implementationThe modified fault-tolerant RAxML-NG code is available under GNU GPL at https://github.com/lukashuebner/ft-raxml-ng.Supplementary information Supplementary data are available at Bioinformatics online.

Efficient and Accurate Selection of Optimal Collective Communication Algorithms Using Analytical Performance Modeling

The performance of collective operations has been a critical issue since the advent of Message Passing Interface (MPI). Many algorithms have been proposed for each MPI collective operation but none of them proved optimal in all situations. Different algorithms demonstrate superior performance depending on the platform, the message size, the number of processes, etc. MPI implementations perform the selection of the collective algorithm empirically, executing a simple runtime decision function. While efficient, this approach does not guarantee the optimal selection. As a more accurate but equally efficient alternative, the use of analytical performance models of collective algorithms for the selection process was proposed and studied. Unfortunately, the previous attempts in this direction have not been successful. We revisit the analytical model-based approach and propose two innovations that significantly improve the selective accuracy of analytical models: (1) We derive analytical models from the code implementing the algorithms rather than from their high-level mathematical definitions. This results in more detailed and relevant models. (2) We estimate model parameters separately for each collective algorithm and include the execution of this algorithm in the corresponding communication experiment. We experimentally demonstrate the accuracy and efficiency of our approach using Open MPI broadcast and gather algorithms and two different Grid'5000 clusters and one supercomputer.

Parallel implementation of a numerical method for solving a three-dimensional transport equation for a mesoscale meteorological model

Abstract A parallel algorithm for numerical solution of a generalized unsteady 3-dimensional advection-diffusion equation for a mesoscale meteorological model is considered. Efficiency of parallel implementations of the solver with Message Passing Interface (MPI), Open Multi-Processing (OpenMP), and NVidia Compute Unified Device Architecture (CUDA) technologies was compared. It was shown that on the mesh of 25625632 nodes the speedup of the program reaches 18 for OpenMP and MPI implementations and 38 for CUDA.

CPL library — A minimal framework for coupled particle and continuum simulation

We present an open-source library for coupling particle codes, such as molecular dynamics (MD) or the discrete element method (DEM), and grid based computational fluid dynamics (CFD). The application is focused on domain decomposition coupling, where a particle and continuum software model different parts of a single simulation domain with information exchange. This focus allows a simple library to be developed, with core mapping and communication handled by just four functions. Emphasis is on scaling on supercomputers, a tested cross-language library, deployment with containers and well-documented simple examples. Building on this core, a template is provided to facilitate the user development of common features for coupling, such as averaging routines and functions to apply constraint forces. The interface code for LAMMPS and OpenFOAM is provided to both include molecular detail in a continuum solver and model fluids flowing through a granular system. Two novel development features are highlighted which will be useful in the development of the next generation of multi-scale software: (i) The division of coupled code into a smaller blocks with testing over a range of processor topologies. (ii) The use of coupled mocking to facilitate coverage of various parts of the code and allow rapid prototyping. These two features aim to help users develop coupled models in a test-driven manner and focus on the physics of the problem instead of just software development. All presented code is open-source with detailed documentation on the dedicated website (cpl-library.org) permitting useful aspects to be evaluated and adopted in other projects. Program summaryProgram Title:CPLLIBRARYProgram Files doi:http://dx.doi.org/10.17632/9dh8w97d2x.1Licensing provisions: GPLv3Programming languages: Fortran/C/C++/PythonExternal routines/libraries: Message Passing Interface (MPI)Nature of problem: Coupling of particle and continuum software to enable simulations not possible with either code alone. In particular, handling communication and interaction of computational fluid dynamics (CFD) software with either molecular dynamics (MD) or discrete element method (DEM) solvers on high performance computing (HPC) platforms.Solution method: A shared library with a minimal set of functions to enable coupling, together with an entire infrastructure to facilitate development of validated coupled software. This includes a minimal Python interface to encourage mock testing, libraries to help develop coupled tools along with pre-coupled examples including OpenFOAM, LAMMPS, Flowmol (Smith, 2014) and interactive plotting using wxPython and matplotlib.Unusual features: Minimal interface with simple setup. A CPL_Mocks framework to facilitate debugging and test driven development. Communication established between independent executables using MPI_Open_Port, reducing required changes to core source code. Coupled codes track the git repository with validation through continuous integration testing and deployment on DockerHub. All communication based on MPI_Cart and MPI_Graph so optimisation is possible through MPI implementation on supercomputers.

The MPI_T events interface: An early evaluation and overview of the interface

Understanding the behavior of parallel applications that use the Message Passing Interface (MPI) is critical for optimizing communication performance. Performance tools for MPI currently rely on the PMPI Profiling Interface or the MPI Tool Information Interface, MPI_T, for portably collecting information for performance measurement and analysis. While tools using these interfaces have proven to be extremely valuable for performance tuning, these interfaces only provide synchronous information, i.e., when an MPI or an MPI_T function is called. There is currently no option for collecting information about asynchronous events from within the MPI library. In this work we propose a callback-driven interface for event notification from MPI implementations. Our approach is integrated in the existing MPI_T interface and provides a portable API for tools to discover and register for events of interest. We implement our MPI_T Events interface in Open MPI and demonstrate its functionality and usability with a small logging tool (MEL) as well as an early integration into the comprehensive measurement infrastructure Score-P.

The application of Hadoop in structural bioinformatics.

The paper reviews the use of the Hadoop platform in structural bioinformatics applications. For structural bioinformatics, Hadoop provides a new framework to analyse large fractions of the Protein Data Bank that is key for high-throughput studies of, for example, protein-ligand docking, clustering of protein-ligand complexes and structural alignment. Specifically we review in the literature a number of implementations using Hadoop of high-throughput analyses and their scalability. We find that these deployments for the most part use known executables called from MapReduce rather than rewriting the algorithms. The scalability exhibits a variable behaviour in comparison with other batch schedulers, particularly as direct comparisons on the same platform are generally not available. Direct comparisons of Hadoop with batch schedulers are absent in the literature but we note there is some evidence that Message Passing Interface implementations scale better than Hadoop. A significant barrier to the use of the Hadoop ecosystem is the difficulty of the interface and configuration of a resource to use Hadoop. This will improve over time as interfaces to Hadoop, e.g. Spark improve, usage of cloud platforms (e.g. Azure and Amazon Web Services (AWS)) increases and standardised approaches such as Workflow Languages (i.e. Workflow Definition Language, Common Workflow Language and Nextflow) are taken up.

TANGO: A high through-put conformation generation and semiempirical method-based optimization tool for ligand molecules.

Lead optimization is one of the crucial steps in the drug discovery pipeline. After identifying the lead molecule and obtaining its 2D geometry, understanding the best conformation it would attain in 3D still remains one of the most challenging steps in drug discovery. There have been multiple methods and algorithms that are directed toward achieving best conformation for the lead molecules. TANGO focuses on conformation generation and its optimization using semiempirical energy calculations. The conformation generation is based on torsion angle rotation of the exocyclic bonds. The energy calculations are performed using MOPAC. The unique feature of this tool lies in the implementation of Message Passing Interface (MPI) for conformation generation and semiempirical-based optimization. A well-defined architecture handling the input and output generation has been used. The master and slave approach to handle operations involved in torsion angle rotation and energy calculations has helped in load balancing the process of conformation generation. The benchmarking results suggest that TANGO scales significantly well across eight nodes with each node utilizing 16 cores. This tool may prove to very useful in high throughput generation of semiempirically optimized small molecule conformations. The use of semiempirical methods for optimization generates a conformational ensemble thereby helping to obtain stable and alternate stable conformers for a given ligand molecule. © 2018 Wiley Periodicals, Inc.

OpenMP and MPI implementations of an elasto-viscoplastic fast Fourier transform-based micromechanical solver for fast crystal plasticity modeling

We explore several parallel implementations of an elasto-viscoplastic fast Fourier transform (EVPFFT) model using Message Passing Interface (MPI), OpenMP, and a hybrid of MPI and OpenMP to efficiently predict micromechanical response of polycrystals. Performance studies using EVPFFT are performed based on domain decomposition over voxels of a periodic cell, which is a representative volume element (RVE) of polycrystalline copper. We begin by parallelizing the computationally intensive Newton–Raphson (NR) single crystal solver within EVPFFT. Next, we compare the performance of the serial and parallel FFTW (Fastest Fourier Transform in the West) using OpenMP (OpenMP-FFTW) and MPI (MPI-FFTW) with the original Numerical Recipes-based FOURN routine within EVPFFT. In the parallel environment, we find that the FFT calculations are best performed using the MPI version of FFTW. Finally, the remainder of the code, except read/write subroutines, is parallelized. Significant speedups of the original EVPFFT model are achieved using MPI on shared memory multicore workstations. Furthermore, results achieved on a distributed memory Cray supercomputer show promising strong and weak scalability and in some cases even super scalability for the single crystal NR solver in EVPFFT. MPI-FFTW also scales perfectly for microstructure RVEs larger than 643 FFT voxels. For example, the MPI-EVPFFT parallel version of the code accelerates the simulations for approximately two orders of magnitude using 64 cores over the old serial code for an RVE size of 1283. The parallel EVPFFT code developed in this work can run massive voxel-based microstructural RVEs taking the advantages of thousands of logical cores provided by more advanced clusters.

OpenMP Parallelization Strategies for a Discontinuous Galerkin Solver

This paper aims to report on the open multi-processing (OpenMP) parallel implementation of a fully unstructured high-order discontinuous Galerkin (DG) solver for computational fluid dynamics and computational aeroacoustics applications. Even if the use of OpenMP paradigm is confined to shared memory systems, it has some advantages over the use of the message passing interface (MPI) library, and getting the best of this approach potentially improves the parallel efficiency of codes running on clusters of multi-core nodes. While with MPI the use of a domain decomposition algorithm is almost unavoidable, the OpenMP shared memory context offers several opportunities. Three strategies, here optimised for a DG solver, are presented and compared: the first refers to a customization of a colouring approach, the second mimics an MPI implementation in the OpenMP context, while the third method is somehow half way between the previous two. The numerical tests performed on both inviscid and viscous test cases indicate that, thanks to the compactness of the DG discretization, all the code versions perform quite satisfactory. In particular, the domain decomposition algorithm reaches the highest level of parallel efficiency at low computational loads while the colouring approach excels at larger computational loads and it can be easily implemented within an existing MPI code. Moreover, colouring is very well suited to deal with hardware accelerators, an opportunity given by the OpenMP 4.0 standard. Finally, the performance gain observed in using a hybrid MPI/OpenMP version of the DG code on high performance computing facilities is demonstrated.

Aero-structural optimization of wind turbine blades using a reduced set of design load cases including turbulence

Modern wind turbine aero-structural blade design codes generally use a smaller fraction of the full design load base (DLB) or neglect turbulent inflow as defined by the International Electrotechnical Commission standards. The current article describes an automated blade design optimization method based on surrogate modeling that includes a very large number of design load cases (DLCs) including turbulence. In the present work, 325 DLCs representative of the full DLB are selected based on the message-passing-interface (MPI) limitations in Matlab. Other methods are currently being investigated, e.g. a Python MPI implementation, to overcome the limitations in Matlab MPI and ultimately achieve a full DLB optimization framework. The reduced DLB and the annual energy production are computed using the state-of-the-art aero-servo-elastic tool HAWC2. Furthermore, some of the interior dimensions of the blade structure are optimized using the finite-element based cross-sectional analysis tool BECAS. The optimization framework is applied to redesign the NREL 5 MW wind turbine blade to obtain improvements in rotor performance and blade weight.

Numerical Modeling of 3D Seismic Wave Propagation around Yogyakarta, the Southern Part of Central Java, Indonesia, Using Spectral-Element Method on MPI-GPU Cluster

A strong tectonic earthquake with a magnitude of 5.9 Richter scale has been occurred in Yogyakarta and Central Java on May 26, 2006. The earthquake has caused severe damage in Yogyakarta and the southern part of Central Java, Indonesia. The understanding of seismic response of earthquake among ground shaking and the level of building damage is important. We present numerical modeling of 3D seismic wave propagation around Yogyakarta and the southern part of Central Java using spectral-element method on MPI-GPU (Graphics Processing Unit) computer cluster to observe its seismic response due to the earthquake. The homogeneous 3D realistic model is generated with detailed topography surface. The influences of free surface topography and layer discontinuity of the 3D model among the seismic response are observed. The seismic wave field is discretized using spectral-element method. The spectral-element method is solved on a mesh of hexahedral elements that is adapted to the free surface topography and the internal discontinuity of the model. To increase the data processing capabilities, the simulation is performed on a GPU cluster with implementation of MPI (Message Passing Interface).

Performance portability in reverse time migration and seismic modelling via OpenACC

Heterogeneity among the computational resources within a single machine has significantly increased in high performance computing to exploit the tremendous potential of graphics processing units (GPUs). Portability in terms of code development and performance has been a challenge due to major differences between GPU programming and memory models from one side and conventional central processing units (CPUs) from another side. Performance characteristics of compilers and processors also vary between machines. Emerging high-level directive-based programming models such as OpenACC has been proposed to target this challenge. In this work, we develop OpenACC implementations for both seismic modelling and reverse time migration algorithms that solve the isotropic, acoustic, and elastic wave equations. We employ OpenACC to take advantage of the computational power of two Nvidia GPU cards: (1) M2090 and (2) K40, residing in IBM and CRAY XC30 clusters respectively. We also explore the main aspects of hybridization seismic modelling and reverse time migration by implementing an Message Passing Interface (MPI)+OpenACC approach. We expose various mapping techniques to develop a portable code that maximizes performance regardless of compiler or platform. Depending on the intensity of the computations, different propagators exhibited different speedup behaviours against a full socket CPU MPI implementation. A performance enhancement of ~10× was obtained, when the acoustic model was ported to a single GPU, compared with a 1.7× speedup obtained using the isotropic model. Our MPI+OpenACC implementation of reverse time migration and seismic modelling shows promising scaling when multiple GPUs were used.