Message Passing Interface Research Articles

Bayesian Phylogenetic Analysis on Multi-Core Compute Architectures: Implementation and Evaluation of BEAGLE in RevBayes With MPI.

Phylogenies are central to many research areas in biology and commonly estimated using likelihood-based methods. Unfortunately, any likelihood-based method, including Bayesian inference, can be restrictively slow for large datasets-with many taxa and/or many sites in the sequence alignment-or complex substitutions models. The primary limiting factor when using large datasets and/or complex models in probabilistic phylogenetic analyses is the likelihood calculation, which dominates the total computation time. To address this bottleneck, we incorporated the high-performance phylogenetic library BEAGLE into RevBayes, which enables multi-threading on multi-core CPUs and GPUs, as well as hardware specific vectorized instructions for faster likelihood calculations. Our new implementation of RevBayes+BEAGLE retains the flexibility and dynamic nature that users expect from vanilla RevBayes. In addition, we implemented native parallelization within RevBayes without an external library using the message passing interface (MPI); RevBayes+MPI. We evaluated our new implementation of RevBayes+BEAGLE using multi-threading on CPUs and 2 different powerful GPUs (NVidia Titan V and NVIDIA A100) against our native implementation of RevBayes+MPI. We found good improvements in speedup when multiple cores were used, with up to 20-fold speedup when using multiple CPU cores and over 90-fold speedup when using multiple GPU cores. The improvement depended on the data type used, DNA or amino acids, and the size of the alignment, but less on the size of the tree. We additionally investigated the cost of rescaling partial likelihoods to avoid numerical underflow and showed that unnecessarily frequent and inefficient rescaling can increase runtimes up to 4-fold. Finally, we presented and compared a new approach to store partial likelihoods on branches instead of nodes that can speed up computations up to 1.7 times but comes at twice the memory requirements.

Structure-exploiting interior-point solver for high-dimensional entropy-sparsified regression learning

The solution of high-dimensional nonlinear regression problems through standard machine learning approaches often relies on first-order information, due to the numerical and memory challenges arising from the computation of the Hessian matrix and of the higher-order derivatives. While this scenario seems not favorable to second-order methods, here we show that an efficient and modular structure-exploiting interior-point solver can be successfully applied to the recently introduced class of entropy-based methods for regression learning. Specifically, by exploiting the favorable structure of the problem and of the Hessian matrix, we suggest a robust solution strategy based on explicit low-rank updates combined with an iterative Symmetric Quasi-Minimal Residual (SQMR) algorithm to solve the underlying system of linear equations. The results show that the proposed structure-exploiting solver – which relies on the hybrid parallelism and distributed-memory computing paradigm – allows a significant solution time speed-up with respect to a naive solution strategy. Furthermore, through an adequate use of the Message Passing Interface (MPI) and of Open Multi-Processing (OpenMP), the proposed solver enables the solution of large-scale problems on high-performance computing architectures consisting of thousands of compute nodes. The accompanying detailed convergence and performance analyses demonstrate both numerical robustness and high-performance capabilities for increasingly high-dimensional problems.

GLOBGM v1.0: a parallel implementation of a 30 arcsec PCR-GLOBWB-MODFLOW global-scale groundwater model

Abstract. We discuss the various performance aspects of parallelizing our transient global-scale groundwater model at 30′′ resolution (30 arcsec; ∼ 1 km at the Equator) on large distributed memory parallel clusters. This model, referred to as GLOBGM, is the successor of our 5′ (5 arcmin; ∼ 10 km at the Equator) PCR-GLOBWB 2 (PCRaster Global Water Balance model) groundwater model, based on MODFLOW having two model layers. The current version of GLOBGM (v1.0) used in this study also has two model layers, is uncalibrated, and uses available 30′′ PCR-GLOBWB data. Increasing the model resolution from 5′ to 30′′ creates challenges, including increased runtime, memory usage, and data storage that exceed the capacity of a single computer. We show that our parallelization tackles these problems with relatively low parallel hardware requirements to meet the needs of users or modelers who do not have exclusive access to hundreds or thousands of nodes within a supercomputer. For our simulation, we use unstructured grids and a prototype version of MODFLOW 6 that we have parallelized using the message-passing interface. We construct independent unstructured grids with a total of 278 million active cells to cancel all redundant sea and land cells, while satisfying all necessary boundary conditions, and distribute them over three continental-scale groundwater models (168 million – Afro–Eurasia; 77 million – the Americas; 16 million – Australia) and one remaining model for the smaller islands (17 million). Each of the four groundwater models is partitioned into multiple non-overlapping submodels that are tightly coupled within the MODFLOW linear solver, where each submodel is uniquely assigned to one processor core, and associated submodel data are written in parallel during the pre-processing, using data tiles. For balancing the parallel workload in advance, we apply the widely used METIS graph partitioner in two ways: it is straightforwardly applied to all (lateral) model grid cells, and it is applied in an area-based manner to HydroBASINS catchments that are assigned to submodels for pre-sorting to a future coupling with surface water. We consider an experiment for simulating the years 1958–2015 with daily time steps and monthly input, including a 20-year spin-up, on the Dutch national supercomputer Snellius. Given that the serial simulation would require ∼ 4.5 months of runtime, we set a hypothetical target of a maximum of 16 h of simulation runtime. We show that 12 nodes (32 cores per node; 384 cores in total) are sufficient to achieve this target, resulting in a speedup of 138 for the largest Afro–Eurasia model when using 7 nodes (224 cores) in parallel. A limited evaluation of the model output using the United States Geological Survey (USGS) National Water Information System (NWIS) head observations for the contiguous United States was conducted. This showed that increasing the resolution from 5′ to 30′′ results in a significant improvement with GLOBGM for the steady-state simulation when compared to the 5′ PCR-GLOBWB groundwater model. However, results for the transient simulation are quite similar, and there is much room for improvement. Monthly and multi-year total terrestrial water storage anomalies derived from the GLOBGM and PCR-GLOBWB models, however, compared favorably with observations from the GRACE satellite. For the next versions of GLOBGM, further improvements require a more detailed (hydro)geological schematization and better information on the locations, depths, and pumping rates of abstraction wells.

Parallel computation to bidimensional heat equation using MPI/CUDA and FFTW package

In this study, we present a fast algorithm for the numerical solution of the heat equation. The heat equation models the heat diffusion over time and through a given region. We engage a finite difference method to solve this equation numerically. The performance of its parallel implementation is considered using Message Passing Interface (MPI), Compute Unified Device Architecture (CUDA), and time schemes, such as Forward Euler (FE) and Runge-Kutta (RK) methods. The originality of this study is research on parallel implementations of the fourth-order Runge-Kutta method (RK4) for sparse matrices on Graphics Processing Unit (GPU) architecture. The supreme proprietary framework for GPU computing is CUDA, provided by NVIDIA. We will show three metrics through this parallelization to compare the computing performance: time-to-solution, speed-up, and performance. The spectral method is investigated by utilizing the FFTW software library, based on the computation of the fast Fourier transforms (FFT) in parallel and distributed memory architectures. Our CUDA-based FFT, named CUFFT, is performed in platforms, which is a highly optimized FFTW implementation. We will give numerical tests to reveal that this method is up-and-coming for solving the heat equation. The final result demonstrates that CUDA has a significant advantage and performance since the computational cost is tiny compared with the MPI implementation. This vital performance gain is also achieved through careful attention of managing memory communication and access.

Test data generation for covering mutation-based path using MGA for MPI program

Message Passing Interface (MPI) is a communication protocol used for parallel programming in various languages, valued for its reliability and broad applicability. Mutation testing is a software testing method for systematically simulating software faults. However, the significant number of inserted mutation branches in the program escalates the testing cost. To address this issue, we propose the comprehensive PMTDGM framework. The framework first generates mutation-based paths based on the relevancy of mutant branches and the difficulty of mutant branch coverage, followed by the establishment of a multitask model of path coverage. Finally, we employ a multi-population genetic algorithm (MGA) to generate test data. Our experiments, performed on six MPI programs of varying sizes and structures, demonstrate that the mutation-based paths have the small test sets and are easy to cover. Additionally, the multitask model and MGA can significantly improve the efficiency of generating test data and reduce the cost of mutation testing for MPI programs compared to traditional methods.

A GPU-ready pseudo-spectral method for direct numerical simulations of multiphase turbulence

In this work, we detail the GPU-porting of an in-house pseudo-spectral solver tailored towards large-scale simulations of interface-resolved simulation of drop- and bubble-laden turbulent flows. The code relies on direct numerical simulation of the Navier-Stokes equations, used to describe the flow field, coupled with a phase-field method, used to describe the shape, deformation, and topological changes of the interface of the drops or bubbles. The governing equations – Navier-Stokes and Cahn-Hilliard equations – are solved using a pseudo-spectral method that relies on transforming the variables in the wavenumber space. The code targets large-scale simulations of drop- and bubble-laden turbulent flows and relies on a multilevel parallelism. The first level of parallelism relies on the message-passing interface (MPI) and is used on multi-core architectures in CPU-based infrastructures. A second level of parallelism relies on OpenACC directives and cuFFT libraries and is used to accelerate the code execution when GPU-based infrastructures are targeted. The resulting multiphase flow solver can be efficiently executed in heterogeneous computing infrastructures and exhibits a remarkable speed-up when GPUs are employed. Thanks to the modular structure of the code and the use of a directive-based strategy to offload code execution on GPUs, only minor code modifications are required when targeting different computing architectures. This improves code maintenance, version control and the implementation of additional modules or governing equations.

Predicting Software Defects in Hybrid MPI and OpenMP Parallel Programs Using Machine Learning

High-performance computing (HPC) and its supercomputers are essential for solving the most difficult issues in many scientific computing domains. The proliferation of computational resources utilized by HPC systems has resulted in an increase in the associated error rates. As such, modern HPC systems promote a hybrid programming style that integrates the message-passing interface (MPI) and open multi-processing (OpenMP). However, this integration often leads to complex defects, such as deadlocks and race conditions, that are challenging to detect and resolve. This paper presents a novel approach: using machine learning algorithms to predict defects in C++-based systems by employing hybrid MPI and OpenMP models. We focus on employing a balanced dataset to enhance prediction accuracy and reliability. Our study highlights the effectiveness of the support vector machine (SVM) classifier, enhanced with term frequency (TF) and recursive feature elimination (RFE) techniques, which demonstrates superior accuracy and performance in defect prediction when compared to other classifiers. This research contributes significantly to the field by providing a robust method for early defect detection in hybrid programming environments, thereby reducing development time, costs and improving the overall reliability of HPC systems.

A scalable parallel computing method for autonomous platoons

This paper developed a scalable parallel computing method that can be used for platoon simulations and controller validations. A scalable adaptive platooning control law was firstly designed, which accommodates a variety of vehicle-to-vehicle communication topologies. A road vehicle dynamics model that considered the Magic Formula tyre model and suspension dynamics was then derived and validated. The parallel computing method adopted the Message Passing Interface technique to allow fast and scalable simulations. Platoon length changes do not require controller and algorithm changes. An 11-vehicle platoon on a real-world 10 km long road section was simulated. Different localisation sensor errors, communication delays, heterogenous vehicle masses and driving modes were considered. Results show that localisation errors have negligible influences on space errors. Aggressive driving and heterogeneous vehicle masses slightly increase space errors (increases less than 0.23 m). Communication delays are the greatest influencer for space errors. Increases for 15, 45 and 75 ms delays were 0.43, 1.41 and 2.41 m, respectively. It is further shown that parallel computing can improve the computing speed by three times on personal computers and seven to 12 times on workstations.

Program partitioning and deadlock analysis for MPI based on logical clocks

The message passing interface (MPI) has become a standard for programming models in the field of high performance computing. It is of great importance to ensure the reliability of MPI programs by detecting whether there exist errors in them. However, as one of the most common errors in MPI programs, deadlock is difficult to detect due to the non-determinism and the asynchronous communication supported by MPI. Existing approaches mainly focus on detecting deadlocks by traversing all possible execution paths in an MPI program. But in this way the detection efficiency is always limited since the number of execution paths increases exponentially with the number of wildcard receives and processes in the program.In order to alleviate the path explosion problem for single-path MPI programs, we propose a program partitioning approach based on logical clocks to detecting deadlocks. In the approach, the program is first divided into several preliminary partitions based on the matching detection rule. Then to obtain the dependency relationships of partitions, the Binary Lazy Clocks algorithm is raised to mark clocks for communication operations. Based on the clocks, the completion orders of communication operations in each process of the program are tracked. Further, we get the dependency relationships of the preliminary partitions by analyzing these completion orders and merge the preliminary partitions with the dependency relationships for generating the final partitions. Finally, deadlocks are detected by traversing all possible execution paths of each final partition. We have implemented our method in a tool called PDMPI and performed experimental evaluation on 14 programs. The experimental results indicate that PDMPI is more effective for detecting deadlocks in MPI programs than two most related tools ISP and SAMPI, especially in programs with numerous interleavings.

Speed Up of Volumetric Non-Local Transform-Domain Filter Utilising HPC Architecture.

This paper presents a parallel implementation of a non-local transform-domain filter (BM4D). The effectiveness of the parallel implementation is demonstrated by denoising image series from computed tomography (CT) and magnetic resonance imaging (MRI). The basic idea of the filter is based on grouping and filtering similar data within the image. Due to the high level of similarity and data redundancy, the filter can provide even better denoising quality than current extensively used approaches based on deep learning (DL). In BM4D, cubes of voxels named patches are the essential image elements for filtering. Using voxels instead of pixels means that the area for searching similar patches is large. Because of this and the application of multi-dimensional transformations, the computation time of the filter is exceptionally long. The original implementation of BM4D is only single-threaded. We provide a parallel version of the filter that supports multi-core and many-core processors and scales on such versatile hardware resources, typical for high-performance computing clusters, even if they are concurrently used for the task. Our algorithm uses hybrid parallelisation that combines open multi-processing (OpenMP) and message passing interface (MPI) technologies and provides up to 283× speedup, which is a 99.65% reduction in processing time compared to the sequential version of the algorithm. In denoising quality, the method performs considerably better than recent DL methods on the data type that these methods have yet to be trained on.

Stochastic Gradient Descent for matrix completion: Hybrid parallelization on shared- and distributed-memory systems

The purpose of this study is to investigate the hybrid parallelization of the Stochastic Gradient Descent (SGD) algorithm for solving the matrix completion problem on a high-performance computing platform. We propose a hybrid parallel decentralized SGD framework with asynchronous inter-process communication and a novel flexible partitioning scheme to attain scalability up to hundreds of processors. We utilize Message Passing Interface (MPI) for inter-node communication and POSIX threads for intra-node parallelism. We tested our method by using different real-world benchmark datasets. Experimental results on a hybrid parallel architecture showed that, compared to the state-of-the-art, the proposed algorithm achieves 6× higher throughput on sparse datasets, while it achieves comparable throughput on relatively dense datasets.

Gaussian approximation potentials: Theory, software implementation and application examples.

Gaussian Approximation Potentials (GAPs) are a class of Machine Learned Interatomic Potentials routinely used to model materials and molecular systems on the atomic scale. The software implementation provides the means for both fitting models using abinitio data and using the resulting potentials in atomic simulations. Details of the GAP theory, algorithms and software are presented, together with detailed usage examples to help new and existing users. We review some recent developments to the GAP framework, including Message Passing Interface parallelisation of the fitting code enabling its use on thousands of central processing unit cores and compression of descriptors to eliminate the poor scaling with the number of different chemical elements.

An Architecture for a Tri-Programming Model-Based Parallel Hybrid Testing Tool

As the development of high-performance computing (HPC) is growing, exascale computing is on the horizon. Therefore, it is imperative to develop parallel systems, such as graphics processing units (GPUs) and programming models, that can effectively utilise the powerful processing resources of exascale computing. A tri-level programming model comprising message passing interface (MPI), compute unified device architecture (CUDA), and open multi-processing (OpenMP) models may significantly enhance the parallelism, performance, productivity, and programmability of the heterogeneous architecture. However, the use of multiple programming models often leads to unexpected errors and behaviours during run-time. It is also difficult to detect such errors in high-level parallel programming languages. Therefore, this present study proposes a parallel hybrid testing tool that employs both static and dynamic testing techniques to address this issue. The proposed tool was designed to identify the run-time errors of C++ and MPI + OpenMP + CUDA systems by analysing the source code during run-time, thereby optimising the testing process and ensuring comprehensive error detection. The proposed tool was able to identify and categorise the run-time errors of tri-level programming models. This highlights the need for a parallel testing tool that is specifically designed for tri-level MPI + OpenMP + CUDA programming models. As contemporary parallel testing tools cannot, at present, be used to test software applications produced using tri-level MPI + OpenMP + CUDA programming models, this present study proposes the architecture of a parallel testing tool to detect run-time errors in tri-level MPI + OpenMP + CUDA programming models.

Finding Bottlenecks in Message Passing Interface Programs by Scalable Critical Path Analysis

Bottlenecks and imbalance in parallel programs can significantly affect performance of parallel execution. Finding these bottlenecks is a key issue in performance analysis of MPI programs especially on a large scale. One of the ways to discover bottlenecks is to analyze the critical path of the parallel program: the longest execution path in the program activity graph. There are a number of methods of finding the critical path; however, most of them suffer a performance drop when scaled. In this paper, we analyze several methods of critical path finding based on classical Dijkstra and Delta-stepping algorithms along with the proposed algorithm based on topological sorting. Corresponding algorithms for each approach are presented including additional enhancements for increasing performance. The implementation of the algorithms and resulting performance for several benchmark applications (NAS Parallel Benchmarks, CP2K, OpenFOAM, LAMMPS, and MiniFE) are analyzed and discussed.

An efficient watermarking algorithm for digital audio data in security applications

Transform-domain audio watermarking systems are more robust than time-domain systems. However, the main weakness of these systems is their high computational cost, especially for long-duration audio signals. Therefore, they are not desirable for real-time security applications where speed is a critical factor. In this paper, we propose a fast watermarking system for audio signals operating in the hybrid transform domain formed by the fractional Charlier transform (FrCT) and the dual-tree complex wavelet transform (DTCWT). The central idea of the proposed algorithm is to parallelize the intensive and repetitive steps in the audio watermarking system and then implement them simultaneously on the available physical cores on an embedded systems cluster. In order to have a low power consumption and a low-cost cluster with a large number of physical cores, four Raspberry Pis 4B are used where the communication between them is ensured using the Message Passing Interface (MPI). The adopted Raspberry Pi cluster is also characterized by its portability and mobility, which are required in watermarking-based smart city applications. In addition to its resistance to any possible manipulation (intentional or unintentional), high payload capacity, and high imperceptibility, the proposed parallel system presents a temporal improvement of about 70%, 80%, and 90% using 4, 8, and 16 physical cores of the adopted cluster, respectively.

A massive MPI parallel framework of smoothed particle hydrodynamics with optimized memory management for extreme mechanics problems

The dynamic failure process of structures under extreme loadings is very common in many fields of engineering and science. The smoothed particle hydrodynamics (SPH) method offers inherent benefits in dealing with complex interfaces and large material deformations in extreme mechanics problems. However, SPH simulations for 3D engineering applications are time-consuming. To address this issue, we introduce MPI (Message Passing Interface) in our SPH scheme to reduce computational time. Some optimizations are adopted to ensure the massive computation of the SPH method. In particular, an optimized memory management strategy is developed to control the memory footprint. With the present MPI-based massive parallelization of the SPH method, several validation examples are tested and analyzed. By comparing the present numerical results with the reference data, the dynamic failure process of complex structures subjected to extreme loadings like explosive and impact loadings can be well captured. A large number of particles, up to 2.04 billion, are adopted in the present simulations. The scaling tests show that the scalability of the massively parallel SPH program achieves a maximum parallel efficiency of 97% on 10020 CPU cores.

Two different parallel approaches for a hybrid fractional order Coronavirus model

In this paper, two different parallel approaches for a hybrid fractional order Coronavirus (2019-nCov) mathematical model are presented. Both parallel approaches are implemented using Julia high level language. Parallel algorithm implementations are developed for the HPC cluster using Message Passing Interface (MPI) technology and general-purpose computing on GPUs (GPGPU) using Compute Unified Device Architecture (CUDA) based on hardware environments. The algorithm implementation are used to solve the real-world problem of the hybrid fractional order Coronavirus (2019-nCov) mathematical model and to study the parallel efficiency. The introduced hybrid fractional order derivative is defined as a linear combination of the integral of Riemann-Liouville and the fractional order Caputo derivative. A parallel algorithm is designed based on the predictor-corrector method with the discretization of the Caputo proportional constant fractional hybrid operator for solving the model problem numerically. Simulation results show that, both the new parallel approaches achieve significant efficiency.

Scalable simulation of coupled adsorption and transport of methane in confined complex porous media with density preconditioning

The growing significance of shales and tight formations in the transition to less carbon-intensive and clean energy drives the research endeavor to understand the physics of gas flow within these systems. However, shales are composed of massively heterogeneous physical and chemical features. Most nano-sized pores connect to millimeter-scale fractures, leading to multiscale transport. These nano-scale pore throats demonstrate non-classical flow behavior, such as non-negligible slip velocities and adsorbed gas layers at the boundary. As a result, classical computational fluid dynamics models do not capture the physics. In this work, we develop a coupling scheme for the multiple-relaxation-time (MRT) lattice Boltzmann (LB) method that integrates the Peng-Robinson equation of state into a pseudo-potential interaction model to capture the physics of methane flow in irregular networks of channels that represent nano-scale porous media. We use atomistic simulations to calibrate and validate our model in slit nano-channels. We propose a preconditioning scheme to initialize the coupled transport and adsorption simulation of methane in complex porous media. The results of this implementation of LB agree with Direct Simulation Monte Carlo (DSMC) and Molecular Dynamics (MD) simulations. We then scale up the LB implementation through vectorization and indirect addressing. We parallelize it using Message Passing Interface (MPI) and OpenMP frameworks to simulate transport and adsorption in complex media with a million lattices. We analyze the differences between coupled and transport-only simulations in two case studies and show that considering phase behavior, i.e., adsorption, can significantly change the flow behavior. This work constitutes an important step towards bridging the gap between molecular flow and system-scale behavior of complex disordered porous media.

Numerical investigation of coaxial turbulent jet

The paper presents a numerical study of a compressible and turbulent coaxial jet, using a Large Eddy Simulation type approach. This study focuses on the implementation of a high-performance and modern numerical method to meet the requirements in terms of results accuracy and calculation cost. The code was parallelized using the Message Passing Interface 'MPI' library. Inviscid fluxes are evaluated using a new linearization solver for the equations characteristic of approximate Riemann problem and the solution has been advanced over time using an explicit two-step McCormack method. The purpose of this study is to understand the initial stage of the transition process from laminar to turbulent flow in coaxial jets. Therefore, it has been determined that the first step in the transition process from laminar to turbulent flow in coaxial jets is initiated by the development of Helmholtz (primary vortices) in sheer layers, while three-dimensionalization turbulence is initiated in the second step with the emergence of secondary vortices. The transverse evolution of Reynolds tensors showed that the axial Reynolds tensor contribution is associated with primary instability while the transversal Reynolds tensor contribution is associated with secondary instability. Flow self-similarity state is obtained in fully developed turbulence region. Axi-symmetric mode predominates on each shear layer in the initial flow region. The results obtained are in good agreement with experimental results similar to this study's simulated jet.

Comparing the Performance of Julia on CPUs versus GPUs and Julia-MPI versus Fortran-MPI: a case study with MPAS-Ocean (Version 7.1)

Abstract. Some programming languages are easy to develop at the cost of slow execution, while others are fast at runtime but much more difficult to write. Julia is a programming language that aims to be the best of both worlds – a development and production language at the same time. To test Julia's utility in scientific high-performance computing (HPC), we built an unstructured-mesh shallow water model in Julia and compared it against an established Fortran-MPI ocean model, the Model for Prediction Across Scales–Ocean (MPAS-Ocean), as well as a Python shallow water code. Three versions of the Julia shallow water code were created: for single-core CPU, graphics processing unit (GPU), and Message Passing Interface (MPI) CPU clusters. Comparing identical simulations revealed that our first version of the Julia model was 13 times faster than Python using NumPy, where both used an unthreaded single-core CPU. Further Julia optimizations, including static typing and removing implicit memory allocations, provided an additional 10–20× speed-up of the single-core CPU Julia model. The GPU-accelerated Julia code was almost identical in terms of performance to the MPI parallelized code on 64 processes, an unexpected result for such different architectures. Parallelized Julia-MPI performance was identical to Fortran-MPI MPAS-Ocean for low processor counts and ranges from 2× faster to 2× slower for higher processor counts. Our experience is that Julia development is fast and convenient for prototyping but that Julia requires further investment and expertise to be competitive with compiled codes. We provide advice on Julia code optimization for HPC systems.