Message Passing Interface Communication Research Articles

Accelerated numerical modeling of shallow water flows with MPI, OpenACC, and GPUs

In this paper, a time-explicit Finite-Volume method is adopted to solve the 2-D shallow water equations on an unstructured triangular mesh, using a two-stage Runge-Kutta integrator and a monotone MUSCL model to achieve second-order accuracy in time and space, respectively. A multi-GPU model is presented that uses the Message Passing Interface (MPI) with OpenACC and uses the METIS library to produce the domain decomposition. A CUDA-aware MPI library (GPUDirect) and overlapped MPI communication with computation are used to improve parallel performance. Two benchmark tests with wet and dry downstream beds are used to test the code's accuracy. Good results were achieved compared to the numerical simulations of published studies. Compared with the multi-CPU version of a 6-core CPU, maximum speedups of 56.18 and 331.51 were obtained using a single GPU and 8 GPUs, respectively. Higher mesh resolution enhances acceleration performance, and the model is applicable to other environmental modeling activities.

Molecular generation strategy and optimization based on A2C reinforcement learning in de novo drug design.

In the field of pharmacochemistry, it is a time-consuming and expensive process for the new drug development. The existing drug design methods face a significant challenge in terms of generation efficiency and quality. In this paper, we proposed a novel molecular generation strategy and optimization based on A2C reinforcement learning. In molecular generation strategy, we adopted transformer-DNN to retain the scaffolds advantages, while accounting for the generated molecules' similarity and internal diversity by dynamic parameter adjustment, further improving the overall quality of molecule generation. In molecular optimization, we introduced heterogeneous parallel supercomputing for large-scale molecular docking based on message passing interface communication technology to rapidly obtain bioactive information, thereby enhancing the efficiency of drug design. Experiments show that our model can generate high-quality molecules with multi-objective properties at a high generation efficiency, with effectiveness and novelty close to 100%. Moreover, we used our method to assist shandong university school of pharmacy to find several candidate drugs molecules of anti-PEDV. The datasets involved in this method and the source code are freely available to academic users at https://github.com/wq-sunshine/MomdTDSRL.git.

Improving strong scalability of electronic structure simulations with reduced overhead of communications

Physically realizable semiconductor electronic devices easily include more than several million atoms even though they are in nanoscale, so electronic structure simulations of huge atomic systems are the key to predict performance characteristics of such devices. The empirical tight-binding (TB) model, which uses a finite set of localized bases for descriptions of a single atom with assumption of nearest-neighbor coupling, has been extensively adopted to solve electronic structures of large-scale atomic systems, and the size-capability of TB simulations has been continuously improved with aids of parallel computing. Recently, we have demonstrated a strong scalability of TB simulations up to 2,500 computing nodes for a model problem consisting of 400 million atoms, where the scalability is limited due to the overhead of collective communications that the core numerical method must involve to get a partial eigenspectrum of the sparse Hamiltonian system matrix. To reduce the cost of Message Passing Interface (MPI) communications and its effect on the end-to-end simulation time, here we design an equivalent transformation of the sparse matrix eigensolver, and employ a core dedicated to MPI communications with dynamic scheduling of threads. Through rigorous benchmark tests against model problems of various sizes, it is confirmed that our method improves the strong scalability of TB simulations by more than 10% in a huge computing environment, by reducing the time-cost of both core computations and collective communications.

Parallel Scalability of Three-Level FROSch Preconditioners to 220000 Cores using the Theta Supercomputer

The parallel performance of the three-level fast and robust overlapping Schwarz (FROSch) preconditioners is investigated for linear elasticity. The FROSch framework is part of the Trilinos software library and contains a parallel implementation of different preconditioners with energy minimizing coarse spaces of generalized Dryja--Smith--Widlund type. The three-level extension is constructed by a recursive application of the FROSch preconditioner to the coarse problem. In this paper, the additional steps in the implementation in order to apply the FROSch preconditioner recursively are described in detail. Furthermore, it is shown that no explicit geometric information is needed in the recursive application of the preconditioner. In particular, the rigid body modes, including the rotations, can be interpolated on the coarse level without additional geometric information. Parallel results for a three-dimensional linear elasticity problem obtained on the Theta supercomputer (Argonne Leadership Computing Facility, Argonne, IL) using up to 220 000 cores are discussed and compared to results obtained on the SuperMUC-NG supercomputer (Leibniz Supercomputing Centre, Garching, Germany). Notably, it can be observed that a hierarchical communication operation in FROSch related to the coarse operator starts to dominate the computing time on Theta, which has a dragonfly interconnect, for 100 000 message passing interface (MPI) ranks or more. The same operation, however, scales well and stays within the order of a second in all experiments performed on SuperMUC-NG, which uses a fat tree network. Using hybrid MPI/OpenMP parallelization, the onset of the MPI communication problem on Theta can be delayed. Further analysis of the performance of FROSch on large supercomputers with dragonfly interconnects will be necessary.

Effects of Using a Memory Stalled Core for Handling MPI Communication Overlapping in the SOR Solver on SX-ACE and SX-Aurora TSUBASA

Modern high-performance computing (HPC) systems consist of a large number of nodes featuring multi-core processors. Many computational fluid dynamics (CFD) codes utilize a Message Passing Interface (MPI) to exploit the potential of such systems. In general, the MPI communication costs increase as the number of MPI processes increases. In this paper, we discuss performance of the code in which a core is used as a dedicated communication core when the core cannot contribute to the performance improvement due to memory-bandwidth limitations. By using the dedicated communication core, the communication operations are overlapped with computation operations, thus enabling highly efficient computation by exploiting the limited memory bandwidth and idle cores. The performance evaluation shows that this code can hide the MPI communication times of 90% on the supercomputer SX-ACE system and 80% on the supercomputer SX-Aurora TSUBASA system, and the performance of the successive over-relaxation (SOR) method is improved by 32% on SX-ACE and 20% on SX-Aurora TSUBASA.

An MPI-based MPSoC Platform in FPGA

The increase in the level of integration of components on silicon became possible to build systems with multiple processors on a single chip. These systems are named MPSoCs (Multi-Processor Systems-on-Chip), and some of them use Networks-on-Chip (NoC) as the communication infrastructure to interconnect their components. This work presents an MPSoC platform designed for performance evaluation of an NoC in FPGA. The platform relies on the use of 32-bit programmable processors and an MPI (Message Passing Interface) communication library, providing a flexible infrastructure for traffic generation and analysis, and enabling the development of parallel applications. For evaluation, we implemented a proof-of-concept instance of the proposed platform in FPGA, and the experimental results show that the software layer of the communication infrastructure dominates communication latency.

An offline framework for high-dimensional ensemble Kalman filters to reduce the time to solution

Abstract. The high computational resources and the time-consuming IO (input/output) are major issues in offline ensemble-based high-dimensional data assimilation systems. Bearing these in mind, this study proposes a sophisticated dynamically running job scheme as well as an innovative parallel IO algorithm to reduce the time to solution of an offline framework for high-dimensional ensemble Kalman filters. The dynamically running job scheme runs as many tasks as possible within a single job to reduce the queuing time and minimize the overhead of starting and/or ending a job. The parallel IO algorithm reads or writes non-overlapping segments of multiple files with an identical structure to reduce the IO times by minimizing the IO competitions and maximizing the overlapping of the MPI (Message Passing Interface) communications with the IO operations. Results based on sensitive experiments show that the proposed parallel IO algorithm can significantly reduce the IO times and have a very good scalability, too. Based on these two advanced techniques, the offline and online modes of ensemble Kalman filters are built based on PDAF (Parallel Data Assimilation Framework) to comprehensively assess their efficiencies. It can be seen from the comparisons between the offline and online modes that the IO time only accounts for a small fraction of the total time with the proposed parallel IO algorithm. The queuing time might be less than the running time in a low-loaded supercomputer such as in an operational context, but the offline mode can be nearly as fast as, if not faster than, the online mode in terms of time to solution. However, the queuing time is dominant and several times larger than the running time in a high-loaded supercomputer. Thus, the offline mode is substantially faster than the online mode in terms of time to solution, especially for large-scale assimilation problems. From this point of view, results suggest that an offline ensemble Kalman filter with an efficient implementation and a high-performance parallel file system should be preferred over its online counterpart for intermittent data assimilation in many situations.

A General Design for a Scalable MPI-GPU Multi-Resolution 2D Numerical Solver

This article presents a multi-GPU implementation of a Finite-Volume solver on a multi-resolution grid. The implementation completely offloads the computation to the GPUs and communications between different GPUs are implemented by means of the Message Passing Interface (MPI) API. Different domain decomposition techniques have been considered and the one based on the Hilbert Space Filling Curves (HSFC) showed optimal scalability. Several optimizations are introduced: One-to-one MPI communications among MPI ranks are completely masked by GPU computations on internal cells and a novel dynamic load balancing algorithm is introduced to minimize the waiting times at global MPI synchronization barriers. Such algorithm adapts the computational load of ranks in response to dynamical changes in the execution time of blocks and in network performances; Its capability to converge to a balanced computation has been empirically shown by numerical experiments. Tests exploit up to 64 GPUs and 83M cells and achieve an efficiency of 90 percent in weak scalability and 85 percent for strong scalability. The framework is general and the results of the article can be ported to a wide range of explicit 2D Partial Differential Equations solvers.

Parallel performance of molecular dynamics trajectory analysis

SummaryThe performance of biomolecular molecular dynamics simulations has steadily increased on modern high‐performance computing resources but acceleration of the analysis of the output trajectories has lagged behind so that analyzing simulations is becoming a bottleneck. To close this gap, we studied the performance of trajectory analysis with message passing interface (MPI) parallelization and the Python MDAnalysis library on three different Extreme Science and Engineering Discovery Environment (XSEDE) supercomputers where trajectories were read from a Lustre parallel file system. Strong scaling performance was impeded by stragglers, MPI processes that were slower than the typical process. Stragglers were less prevalent for compute‐bound workloads, thus pointing to file reading as a bottleneck for scaling. However, a more complicated picture emerged in which both the computation and the data ingestion exhibited close to ideal strong scaling behavior whereas stragglers were primarily caused by either large MPI communication costs or long times to open the single shared trajectory file. We improved overall strong scaling performance by either subfiling (splitting the trajectory into separate files) or MPI‐IO with parallel HDF5 trajectory files. The parallel HDF5 approach resulted in near ideal strong scaling on up to 384 cores (16 nodes), thus reducing trajectory analysis times by two orders of magnitude compared with the serial approach.

MPI-RCDD: A Framework for MPI Runtime Communication Deadlock Detection

The message passing interface (MPI) has become a de facto standard for programming models of high-performance computing, but its rich and flexible interface semantics makes the program easy to generate communication deadlock, which seriously affects the usability of the system. However, the existing detection tools for MPI communication deadlock are not scalable enough to adapt to the continuous expansion of system scale. In this context, we propose a framework for MPI runtime communication deadlock detection, namely MPI-RCDD, which contains three kinds of main mechanisms. Firstly, MPI-RCDD has a message logging protocol that is associated with deadlock detection to ensure that the communication messages required for deadlock analysis are not lost. Secondly, it uses the asynchronous processing thread provided by the MPI to implement the transfer of dependencies between processes, so that multiple processes can participate in deadlock detection simultaneously, thus alleviating the performance bottleneck problem of centralized analysis. In addition, it uses an AND⊕OR model based algorithm named AODA to perform deadlock analysis work. The AODA algorithm combines the advantages of both timeout-based and dependency-based deadlock analysis approaches, and allows the processes in the timeout state to search for a deadlock circle or knot in the process of dependency transfer. Further, the AODA algorithm cannot lead to false positives and can represent the source of the deadlock accurately. The experimental results on typical MPI communication deadlock benchmarks such as Umpire Test Suit demonstrate the capability of MPI-RCDD. Additionally, the experiments on the NPB benchmarks obtain the satisfying performance cost, which show that the MPI-RCDD has strong scalability.

Large-Scale Parallel Implementation of Hartree-Fock Exchange Energy on Real-Space Grids Using 3D-Parallel Fast Fourier Transform.

Two types of implementation of the Hartree-Fock (HF) exchange energy were developed with the real-space grid approach for the purpose of achieving high efficiency in the parallel execution of the hybrid exchange functional in the density functional theory. First, a parallel implementation of the three-dimensional fast Fourier transform (FFT), referred to as PFFT, was adapted to solve the Poisson equations for the electrostatic potentials of the densities of the orbital pairs. In the other approach, the Poisson equations were solved iteratively through the conjugate gradient (CG) procedures where the operation of Laplacian was parallelized by the domain decomposition scheme. Comparison of the parallel performances for the exchange energy calculation was made between these two approaches, and it was revealed that the calculation with the FFT method is faster than that with CG. The method with FFT is more advantageous than CG because a larger bandwidth can be made available in the collective message passing interface communication associated with the parallel execution of FFT. We also implemented the projection operator to circumvent the laborious calculation of the exchange energy at every self-consistent field step, which made a significant contribution to expedite the convergence. To assess the accuracy of our implementation, the association energies of a hydrated ion were computed, which showed excellent agreement with those given by the Gaussian 09 program employing sophisticated basis sets.

Distance-learning For Approximate Bayesian Computation To Model a Volcanic Eruption

Approximate Bayesian computation (ABC) provides us with a way to infer parameters of models, for which the likelihood function is not available, from an observation. Using ABC, which depends on many simulations from the considered model, we develop an inferential framework to learn parameters of a stochastic numerical simulator of volcanic eruption. Moreover, the model itself is parallelized using Message Passing Interface (MPI). Thus, we develop a nested-parallelized MPI communicator to handle the expensive numerical model with ABC algorithms. ABC usually relies on summary statistics of the data in order to measure the discrepancy model output and observation. However, informative summary statistics cannot be found for the considered model. We therefore develop a technique to learn a distance between model outputs based on deep metric-learning. We use this framework to learn the plume characteristics (eg. initial plume velocity) of the volcanic eruption from the tephra deposits collected by field-work associated with the 2450 BP Pululagua (Ecuador) volcanic eruption.

Lattice H-Matrices for Massively Parallel Micromagnetic Simulations of Current-Induced Domain Wall Motion

This article discusses parallel hierarchical-matrices ( $\mathcal {H}$ -matrices) to compute a demagnetizing field, which is the most time-consuming part in the micromagnetic simulation of current-induced domain wall motion (CDWM). Although normal $\mathcal {H}$ -matrices exhibit high efficiencies for small numbers of message passing interface (MPI) processes, the performance rapidly decays due to load imbalance and the MPI communication costs as the number of processes increases. We introduce lattice $\mathcal {H}$ -matrices to improve the parallel scalability, when using a large number of processes. The applicability of lattice $\mathcal {H}$ -matrices to CDWM simulations is confirmed and proper lattice block sizes and process grid shapes of the lattice $\mathcal {H}$ -matrices for $\mathcal {H}$ -matrix-vector products (HMVPs) are investigated using practical data sets. Under appropriate settings, the lattice $\mathcal {H}$ -matrices exhibit almost linear complexity in memory usage and calculation time of HMVPs. Our implementation continues to accelerate at least up to about 3600 MPI processes, even in a small problem with several tens of thousands of unknowns.

A dynamic, unified design for dedicated message matching engines for collective and point-to-point communications

The Message Passing Interface (MPI) libraries use message queues to guarantee correct message ordering between communicating processes. Message queues are in the critical path of MPI communications and thus, the performance of message queue operations can have significant impact on the performance of applications. Collective communications are widely used in MPI applications and they can have considerable impact on generating long message queues. In this paper, we propose a unified message matching mechanism that improves the message queue search time by distinguishing messages coming from point-to-point and collective communications and using a distinct message queue data structure for them. For collective operations, it dynamically profiles the impact of each collective call on message queues during the application runtime and uses this information to adapt the message queue data structure for each collective dynamically. Moreover, we use a partner/non-partner message queue data structure for the messages coming from point-to-point communications. The proposed approach can successfully reduce the queue search time while maintaining scalable memory consumption. The evaluation results show that we can obtain up to 5.5x runtime speedup for applications with long list traversals. Moreover, we can gain up to 15% and 94% queue search time improvement for all elements in applications with short and medium list traversals, respectively.

Optimization of Patterned Surfaces for Improved Superhydrophobicity through Cost-Effective Large-Scale Computations.

The pattern design of superhydrophobic surfaces can be significantly aided by computations that predict the Cassie-Baxter (CB) to Wenzel (W) transition, which is responsible for the break-down of superhydrophobic behavior. We present a computational framework for the optimization of patterned surfaces based on the energy barriers of the CB-W transitions which comprises the following elements: (a) design of structured surface patterns, for example, arrays of pillars, with parameterized geometric features such as size, pitch, slope, and roundness. (b) Computation of the wetting states with a modified Young-Laplace equation that facilitates the introduction of solid/liquid interactions for complex surface patterns and has significantly lower computational cost than other commonly used methods, such as the volume-of-fluid, phase-field, and so forth. (c) Incorporation of the modified Young-Laplace in the simplified string method, allowing the calculation of the minimum energy paths of wetting transitions which, apart from the energy barriers, also reveal the transition mechanisms (CB failure modes). (d) Accommodation of large-scale problems with good parallel performance and scalability on multicore-distributed memory systems using fast iterative solvers and the Message Passing Interface communication protocol. We demonstrate the computational framework with a shape optimization study of inverted conical frustum pillars. The optimization objective function is the resistance to the CB-W transition, which is quantified by the energy barrier-a relatively large energy barrier suggests improved superhydrophobicity. We also report the parallel performance, in terms of parallel speedup for problems ranging from three hundred thousands to 12 million degrees of freedom, solved using up to 40 processing cores.

Conservative Multimoment Transport along Characteristics for Discontinuous Galerkin Methods

A set of algorithms based on characteristic discontinuous Galerkin methods is presented for tracer transport on the sphere. The algorithms are designed to reduce message passing interface communication volume per unit of simulated time relative to current methods generally, and to the spectral element scheme employed by the U.S. Department of Energy's Exascale Earth System Model (E3SM) specifically. Two methods are developed to enforce discrete mass conservation when the transport schemes are coupled to a separate dynamics solver; constrained transport and Jacobian-combined transport. A communication-efficient method is introduced to enforce tracer consistency between the transport scheme and dynamics solver; this method also provides the transport scheme's shape preservation capability. A subset of the algorithms derived here is implemented in E3SM and shown to improve transport performance by a factor of 2.2 for the model's standard configuration with 40 tracers at the strong scaling limit of one element per core.

Developing a parallel density-based implicit solver with mesh deformation in OpenFOAM

Numerical simulations like the aeroelastic computation and aerodynamic shape optimization usually involve moving boundaries, and are always carried out with supersonic compressible flows. In this paper, based on the OpenFOAM platform, we present a parallel density-based implicit solver with mesh deformation to address this kind of problems, and fill the gap for the deficiencies of implicit solvers. The core implementation details of the solver based on OpenFOAM are introduced. The Godunov type schemes in ALE (Arbitrary Lagrangian Eulerian) formulation and the dual-time stepping LU-SGS (Lower-Upper Symmetric Gauss-Seidel) algorithm are applied to solve the Navier-Stokes equations with turbulence model. For dynamic meshes, the novel parallel mesh deformation approach based on the Support Vector Machine is implemented. Four typical cases are tested to demonstrate the validity and parallel efficiency of the proposed solver. The result shows that the scalability of the solver reaches its limit at around 128 MPI (Message Passing Interface) tasks, and the cost of MPI communication is the bottleneck for large-scale simulations. Overall, our solver is applicable for various dynamic mesh compressible problems and all types of grids.

A Minimally Intrusive Low-Memory Approach to Resilience for Existing Transient Solvers

We propose a novel, minimally intrusive approach to adding fault tolerance to existing complex scientific simulation codes, used for addressing a broad range of time-dependent problems on the next generation of supercomputers. Exascale systems have the potential to allow much larger, more accurate and scale-resolving simulations of transient processes than can be performed on current petascale systems. However, with a much larger number of components, exascale computers are expected to suffer a node failure every few minutes. Many existing parallel simulation codes are not tolerant of these failures and existing resilience methodologies would necessitate major modifications or redesign of the application. Our approach combines the proposed user-level failure mitigation extensions to the Message-Passing Interface (MPI), with the concepts of message-logging and remote in-memory checkpointing, to demonstrate how to add scalable resilience to transient solvers. Logging MPI communication reduces the storage requirement of static data, such as finite element operators, and allows a spare MPI process to rebuild these data structures independently of other ranks. Remote in-memory checkpointing avoids disk I/O contention on large parallel filesystems. A prototype implementation is applied to Nektar++, a scalable, production-ready transient simulation framework. Forward-path and recovery-path performance of the resilience algorithm is analysed through experiments using the solver for the incompressible Navier–Stokes equations, and strong scaling of the approach is observed.

SIMPLE adjoint message passing

In the context of numerical simulation a system of partial differential equations is typically solved by discretizing it into a set of linear equations. The non-linear effects in the solution are then approximated by repeatedly discretizing, solving and correcting the equations using some outer iteration scheme. One example for such a scheme is the SIMPLE algorithm. In this paper, we show how we obtained an algorithmic adjoint version of the SIMPLE algorithm in OpenFOAM with particular focus on the adjoint message passing interface communication inside the linear solvers. Due to the sparse matrix storage and specifics of the implementation of the linear solvers, this method requires special attention to the inner workings of said structures and algorithms. We show the feasibility of the chosen method by presenting two case studies. In addition to the SIMPLE algorithm, the presented ideas are applicable to a wide range of algorithms whenever linear systems are embedded into an overlying non-linear context.

UnisonFlow: A Software-Defined Coordination Mechanism for Message-Passing Communication and Computation

Message passing interface (MPI) communication performance is becoming one of the key factors heavily affecting the total performance of data-intensive applications running on computer clusters. Our software-defined networking (SDN)-enhanced MPI improves the performance of communication over interconnects by integrating flexible and dynamic network controllability of SDN into MPI. We have demonstrated that the acceleration of individual MPI communication primitives is feasible through our past work on the SDN-enhanced MPI. However, real-world MPI applications have not benefited from such accelerated communication primitives through our research achievements to date, because each of the distinct network control algorithms designed for various MPI communication primitives cannot be activated and coordinated with the execution of the MPI application. Therefore, this paper proposes UnisonFlow, a software-defined coordination mechanism for the SDN-enhanced MPI that performs network control in synchronization with the execution of applications. An experiment conducted on a real-computer cluster verifies that the interconnect control can be successfully performed in synchronization with the execution of the application. Furthermore, the synchronization is performed with a low overhead and its performance penalty is practically negligible.