MPI Applications Research Articles

Methodology to define a static allocation mapping based on memory access patterns and the Signature of MPI applications in HPC systems

Tools to identify problems and improve MPI applications performance running on an HPC system require information from the application and the system. In this work we will focus on defining a methodology to analyze how memory usage affects an MPI application performance running on this HPC system. This methodology will obtain valid and comparable data based on different memory access patterns that will allow defining key performance values used to characterize the HPC system behavior in front of these access patterns and to characterize the Application Signature behavior, (PAS2P Signature that obtains the representative phases of the MPI application) in front of these same access patterns. With these key performance values, we will be able to detect memory access application problems, suggest improvements and define a mapping policy for this application in this HPC system to improve its performance and to determine limits to these improvements.

Analysis of leakage artifacts and their impact on convergence of algebraic reconstruction in multi-contrast magnetic particle imaging

Objective.Magnetic particle imaging (MPI) is a tracer-based medical imaging modality with great potential due to its high sensitivity, high spatiotemporal resolution, and ability to quantify the tracer concentration. Image reconstruction in MPI is an ill-posed problem, which the use of regularization methods can address. Multi-contrast MPI reconstructs the signal from different tracer materials or environments separately, resulting in multi-channel images that enable quantification of, for example, temperature or viscosity. Single- and multi-contrast MPI reconstructions produce different kinds of artifacts. The objective of this work is threefold: first, to present the concept of multi-contrast specific MPI channel leakage artifacts; second, to ascertain the source of these leakage artifacts; and third, to introduce a method for their reduction.Approach.A definition for leakage artifacts is established, and a quantification method is proposed. A comprehensive analysis is conducted to establish a connection between the properties of the multi-contrast MPI system matrix and the leakage artifacts. Moreover, a two-step measurement and reconstruction method is introduced to reduce channel leakage artifacts between multi-contrast MPI channels.Main results.The severity of these artifacts correlates with the system matrix shape and condition number and depends on the similarity of the corresponding frequency components. Using the proposed two-step method on both semi-simulated and measured data a significant leakage reduction and speed up the convergence of the multi-contrast MPI reconstruction was observed.Significance.The multi-contrast system matrix analysis we conducted is essential for understanding the source of the channel leakage artifacts and finding methods to reduce them. Our proposed two-step method is expected to improve the potential for real-time multi-contrast MPI applications.

Proteo: a framework for the generation and evaluation of malleable MPI applications

Applying malleability to HPC systems can increase their productivity without degrading or even improving the performance of running applications. This paper presents Proteo, a configurable framework that allows to design benchmarks to study the effect of malleability on a system, and also incorporates malleability into a real application. Proteo consists of two modules: SAM allows to emulate the computational behavior of iterative scientific MPI applications, and MaM is able to reconfigure a job during execution, adjusting the number of processes, redistributing data, and resuming execution. An in-depth study of all the possibilities shows that Proteo is able to behave like a real malleable or non-malleable application in the range [0.85, 1.15]. Furthermore, the different methods defined in MaM for process management and data redistribution are analyzed, concluding that asynchronous malleability, where reconfiguration and application execution overlap, results in a 1.15×\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\ imes$$\\end{document} speedup.

Faster and Scalable MPI Applications Launching

Distributed parallel MPI applications are the dominant workload in many high-performance computing systems. While optimizing MPI application execution is a well-studied field, little work has considered optimizing the initial MPI application launching phase, which incurs extensive cross-machine communications and synchronization. The overhead of MPI application launching can be expensive, accounting for over 200 million processor core hours and 15% of the user core time annually on the production Tianhe-2A supercomputer, which will increase as the number of parallel machines used grows. Therefore, it is critical to optimize the MPI application launching process. This paper presents a novel approach to optimizing the MPI application launch. Our approach adopts a location-aware address generation rule to eliminate the need for address exchange and a topology-aware global communication scheme to optimize cross-machine synchronization. We then design a new application launch procedure to support the proposed optimizations to further reduce the pressure of the shared I/O system. Our techniques have been deployed to production in the Tianhe-2A supercomputer and the Next Generation Tianhe Supercomputer. Experimental results show that our approach scales well and outperforms alternative schemes, reducing the MPI application launching time by over 29% with 320K MPI processes.

Combination of time domain-system matrix and x-space methods to reconstruct magnetic particle images with isotropic resolution

Objective. Imaging of superparamagnetic iron oxide nanoparticles based on their non-linear response to alternating magnetic fields shows promise for imaging cells and vasculature in healthy and diseased tissue. Such imaging can be achieved through x-space reconstruction typically along a unidirectional Cartesian trajectory, which rapidly convolutes the particle distribution with a ‘anisotropic blurring’ point spread function (PSF), leading to images with anisotropic resolution. Approach. Here we propose combining the time domine-system matrix and x-space reconstruction methods into a forward model, where the output of the forward model is the PSF-blurred x-space reconstructed image. We then treat the blur as an inverse problem solved by Kaczmarz iteration. Main results. After we have proposed the method optimization, the normal resolution of simulation and device images has been increased from 3.5 mm and 5.25 mm to 1.5 mm and 3.25 mm, which has reached the level in the tangential resolution. Quantitative indicators of image quality such as PSNR and SSIM have also been greatly improved. Significance. Simulation and imaging of real phantoms indicate that our approach provides better isotropic resolution and image quality than the x-space method alone or other methods for removing PSF blur. Using our proposed method to optimize the image quality of x-space reconstructed images using unidirectional Cartesian trajectories, it will promote the clinical application of MPI in the future.

An overview of the Legio fault resilience framework for MPI applications

While the computational power of HPC clusters breached through the exascale milestone, it highlighted the need for some critical features like fault management. The current de facto standard for inter-process communication at scale, MPI, lacks proper fault management functionalities, precluding result production in the presence of faults. Many efforts have already addressed the problem, with the ULFM extension and Reinit proposal being the two works receiving the most attention. While powerful and effective, they require expertise from the user and significantly impact the application code. For this reason, we presented the Legio fault resilience framework, which can introduce graceful degradation properties in MPI applications with minimal changes. This work summarises the development and evolution of Legio, highlighting successful use cases and providing some tips for users who want to leverage it for their executions.

Comparative Analysis of OpenMP and MPI Parallel Computing Implementations in Team Sort Algorithm

Tim Sort is a sorting algorithm that combines Merge Sort and Binary Insertion Sort sorting algorithms. Parallel computing is a computational processing technique in parallel or is divided into several parts and carried out simultaneously. The application of parallel computing to algorithms is called parallelization. The purpose of parallelization is to reduce computational processing time, but not all parallelization can reduce computational processing time. Our research aims to analyse the effect of implementing parallel computing on the processing time of the Tim Sort algorithm. The Team Sort algorithm will be parallelized by dividing the flow or data into several parts, then each sorting and recombining them. The libraries we use are OpenMP and MPI, and tests are carried out using up to 16 core processors and data up to 4194304 numbers. The goal to be achieved by comparing the application of OpenMP and MPI to the Team Sort algorithm is to find out and choose which library is better for the case study, so that when there is a similar case, it can be used as a reference for using the library in solving the problem. The results of research for testing using 16 processor cores and the data used prove that the parallelization of the Sort Team algorithm using OpenMP is better with a speed increase of up to 8.48 times, compared to using MPI with a speed increase of 8.4 times. In addition, the increase in speed and efficiency increases as the amount of data increases. However, the increase in efficiency that is obtained by increasing the processor cores decreases.

An introspection monitoring library to improve MPI communication time

In this paper, we describe how to improve communication time of MPI parallel applications with the use of a library that enables to monitor MPI applications and allows for introspection (the program itself can query the state of the monitoring system). Based on previous work, this library is able to see how collective communications are decomposed into point-to-point messages. It also features monitoring sessions that allow suspending and restarting the monitoring, limiting it to specific portions of the code. Experiments show that the monitoring overhead is very small and that the proposed features allow for dynamic and efficient rank reordering enabling up to 2-time reduction of communication parts of some program.

Near-Lossless MPI Tracing and Proxy Application Autogeneration

Traces of MPI communications are used by many performance analysis and visualization tools. Storing exhaustive traces of large-scale MPI applications is infeasible, however, because of their large volume. Aggregated or lossy MPI traces are smaller but provide much less information. In this paper we present Pilgrim, a near-lossless MPI tracing tool that, by using sophisticated compression techniques, generates small trace files at large scales and incurs only moderate overheads. We perform comprehensive studies of various compression techniques used for storing timestamps associated with each call. This timing information is essential for analysis purposes such as skews study. To demonstrate the usefulness of the detailed information stored by Pilgrim, we present a proxy application generator that can generate proxy apps that preserve original communication patterns from the Pilgrim traces.

MARPLE: программное обеспечение для мультифизического моделирования в задачах сплошных сред

The research code MARPLE was originally created to model high-speed dynamic processes caused by the action of high-intensity energy fluxes on matter. At present, it is a universal tool able to solve various continuum mechanics problems. The implemented physical models are the following: single-fluid two-temperature MHD model of plasma dynamics, including electron-ion energy exchange and generalized Ohm's law; model of electrical and thermal conductivity taking into account the anisotropy in the magnetic field; radiative heat transfer: models pertinent to optically thin as well as optically thick media: techniques for taking into account radiative cooling losses, spectral multigroup diffusion transfer, laser radiation propagation etc.; model of multicomponent flow. Calculations are performed using wide-range equations of state, transport and optical data. The MARPLE code utilises modern computational technologies based on block structured and unstructured meshes consisting of tetrahedral, hexahedral, prismatic elements and their combinations. The solvers implement conservation laws using high-resolution techniques. We apply the physical splitting to solve the governing system. The object-oriented approach to software design is used, as well as methods of object and generic programming (C++ implementation language). Design of computational domains is provided by means of integrated SALOME open source CAD-CAE system. Marple works as MPI application for modern HPC systems. The paper presents examples of problems in plasma dynamics, magnetohydrodynamics, astrophysics, and solid thermomechanics solved by means of the MARPLE code.

Optimized Page Fault Handling During RDMA

Remote Direct Memory Access (RDMA) is widely used in High-Performance Computing (HPC) while making inroads in datacenters and accelerators. State-of-the-art RDMA engines typically do not endure page faults, therefore users are forced to pin their buffers, which complicates the programming model, limits the memory utilization, and moves the pressure to the Network Interface Cards (NICs). In this article we introduce a mechanism for handling dynamic page faults during RDMA, named PART, suitable for emerging processors that also integrate the Network Interface. PART leverages the IOMMU already present in modern processors for translations. PART avoids the pinning overheads, allows any buffer to be used for communication, and enables overlapping page fault handling with serving subsequent RDMA transfers. We implement and optimize PART for a cluster of ARMv8 cores with tightly-coupled network interfaces. Handling a minor page-fault of a small transfer at the destination takes approximately 38 <inline-formula><tex-math notation="LaTeX">$\mu$</tex-math></inline-formula> secs, while there is no performance degradation when running three full MPI applications in 16 nodes and 64 cores. Detailed breakdown uncovers the hardware and system software components of this overhead and was used to further optimize the system. A 4MB RDMA transfer performs 1.46x better over pinning.

Development and prospect of monitoring and prevention methods of icing disaster in China power grid

Icing disaster is one of the most serious threats to power grid security in China. This article provides a brief glance of current China's monitoring and prevention methods of icing disaster (MPI) development status. Despite a late start, China's research and development on MPI have made outstanding achievements, including formation mechanism causing ice disaster, key technologies and characteristics, and anti/de-icing approaches development and application. It is apparent that Chinese scientists and engineers confronted the worldwide problem of anti/de-icing, and put forward a comprehensive solution, which plays a more critical role in icing disaster prevention and reduction of power grid during the past 15 years. The article states that the anti/de-icing devices have been applied successively to severe icing issues. These achievements are leading the world in the development and application of MPI. However, it is only at the extreme ice and snow conditions that can test and verify whether prevention methods are comprehensive and perfect or not. And the authors discovered the application limitations in the existing achievements. The article also discusses the overall development trends of MPI in the areas of key technological and technical breakthroughs and application scenarios etc. These discussions serve as references for MPI's future technological advancement and its ever-expanding applications.

Performance Assessment of Using Docker for Selected MPI Applications in a Parallel Environment Based on Commodity Hardware

In the paper, we perform detailed performance analysis of three parallel MPI applications run in a parallel environment based on commodity hardware, using Docker and bare-metal configurations. The testbed applications are representative of the most typical parallel processing paradigms: master–slave, geometric Single Program Multiple Data (SPMD) as well as divide-and-conquer and feature characteristic computational and communication schemes. We perform analysis selecting best configurations considering various optimization flags for the applications and best execution times and speed-ups in terms of the number of nodes and overhead of the virtualized environment. We have concluded that for the configurations giving the shortest execution times the overheads of Docker versus bare-metal for the applications are as follows: 7.59% for master–slave run using 64 processes (number of physical cores), 15.30% for geometric SPMD run using 128 processes (number of logical cores) and 13.29% for divide-and-conquer run using 256 processes. Finally, we compare results obtained using gcc V9 and V7 compiler versions.

Simulation-based optimization and sensibility analysis of MPI applications: Variability matters

Finely tuning MPI applications and understanding the influence of key parameters (number of processes, granularity, collective operation algorithms, virtual topology, and process placement) is critical to obtain good performance on supercomputers. With the high consumption of running applications at scale, doing so solely to optimize their performance is particularly costly. Having inexpensive but faithful predictions of expected performance could be a great help for researchers and system administrators. The methodology we propose decouples the complexity of the platform, which is captured through statistical models of the performance of its main components (MPI communications, BLAS operations), from the complexity of adaptive applications by emulating the application and skipping regular non-MPI parts of the code. We demonstrate the capability of our method with High-Performance Linpack (HPL), the benchmark used to rank supercomputers in the TOP500, which requires careful tuning. We briefly present (1) how the open-source version of HPL can be slightly modified to allow a fast emulation on a single commodity server at the scale of a supercomputer. Then we present (2) an extensive (in)validation study that compares simulation with real experiments and demonstrates our ability to predict the performance of HPL within a few percent consistently. This study allows us to identify the main modeling pitfalls (e.g., spatial and temporal node variability or network heterogeneity and irregular behavior) that need to be considered. Last, we show (3) how our “surrogate” allows studying several subtle HPL parameter optimization problems while accounting for uncertainty on the platform.

A unified framework to improve the interoperability between HPC and Big Data languages and programming models

One of the most important issues in the path to the convergence of HPC and Big Data is caused by the differences in their software stacks. Despite some research efforts, the interoperability between their programming models and languages is still limited. To deal with this problem we introduce a new computing framework called IgnisHPC, whose main objective is to unify the execution of Big Data and HPC workloads in the same framework. IgnisHPC has native support for multi-language applications using JVM and non-JVM-based languages. Since MPI was used as its backbone technology, IgnisHPC takes advantage of many communication models and network architectures. Moreover, MPI applications can be directly executed in an efficient way in the framework. The main consequence is that users could combine in the same multi-language code HPC tasks (using MPI) with Big Data tasks (using MapReduce operations). The experimental evaluation demonstrates the benefits of our proposal in terms of performance and productivity with respect to other frameworks. IgnisHPC is publicly available for the Big Data and HPC research community.

Model-based selection of optimal MPI broadcast algorithms for multi-core clusters

The performance of collective communication operations determines the overall performance of MPI applications. Different algorithms have been developed and implemented for each MPI collective operation, but none proved superior in all situations. Therefore, MPI implementations have to solve the problem of selecting the optimal algorithm for the collective operation depending on the platform, the number of processes involved, the message size(s), etc. The current solution method is purely empirical.Recently, an alternative solution method using analytical performance models of collective algorithms has been proposed and proved both accurate and efficient for one-process-per-CPU configurations. The method derives the analytical performance models of algorithms from their code implementation rather than from high-level mathematical definitions, and estimates the parameters of the models separately for each algorithm. The method is network and topology oblivious and uses the Hockney model for point-to-point communications. In this paper, we extend that selection method to the case of clusters of multi-core processors, where each core of the platform runs a process of the MPI application.We present the proposed approach using Open MPI broadcast algorithms, and experimentally validate it on three different clusters of multi-core processors, Grisou, Gros and MareNostrum4.

Reconfigurable switches for high performance and flexible MPI collectives

AbstractThere has been much effort in offloading MPI collective operations into hardware. But while NIC‐based collective acceleration is well‐studied, offloading their processing into the switching fabric, despite numerous advantages, has been much more limited. A major problem with fixed logic implementations is that either only a fraction of the possible collective communication is accelerated or that logic is wasted in the applications that do not need a particular capability. Using reconfigurable logic has numerous advantages: exactly the required operations can be implemented; the level of desired performance can be specified; and new, possibly complex, operations can be defined and implemented. We have designed an in‐switch collective accelerator, MPI‐FPGA, and demonstrated its use with seven MPI collectives and over a set of benchmarks and proxy applications (MiniApps). The accelerator uses a novel two‐level switch design containing fully pipelined vectorized aggregation logic units. Essential to this work is providing support for sub‐communicator collectives that enables communicators of arbitrary shape, and that is scalable to large systems. A streaming interface improves the performance for long messages. While this reconfigurable design is generally applicable, we prototype it with an FPGA‐centric cluster. A sample MPI‐FPGA design in a direct network achieves considerable speedups over conventional clusters in the most likely scenarios. We also present results for indirect networks with reconfigurable high‐radix switches and show that this approach is competitive with SHArP technology for the subset of operations that SHArP supports. MPI‐FPGA is fully integrated into MPICH and is transparent to MPI applications.

ANACIN-X: A software framework for studying non-determinism in MPI applications

<h2>Abstract</h2> Non-determinism in MPI applications negatively impacts the debugging and correctness of high performance computing (HPC) and scientific simulations. To help developers and scientists study and understand sources of non-determinism, we present ANACIN-X, a software framework for quantifying point-to-point communication non-determinism in MPI applications as graph kernel distances. ANACIN-X includes three benchmark applications for testing (i.e., message race, algebraic multigrid, and unstructured mesh).

Implementation of the LAMMPS package using T-system with an Open Architecture

Supercomputer applications are usually implemented in the C, C++, and Fortran programming languages using different versions of the Message Passing Interface library. The "T-system" project (OpenTS) studies the issues of automatic dynamic parallelization of programs. In practical terms, the implementation of applications in a mixed (hybrid) style is relevant, when one part of the application is written in the paradigm of automatic dynamic parallelization of programs and does not use any primitives of the MPI library, and the other part of it is written using the Message Passing Interface library. In this case, the library is used, which is a part of the T-system and is called DMPI (Dynamic Message Passing Interface). In this way, it is necessary to evaluate the effectiveness of the MPI implementation available in the T-system. The purpose of this work is to examine the effectiveness of DMPI implementation in the T-system. In a classic MPI application, 0% of the code is implemented using automatic dynamic parallelization of programs and 100% of the code is implemented in the form of a regular Message Passing Interface program. For comparative analysis, at the beginning the code is executed on the standard Message Passing Interface, for which it was originally written, and then it is executed using the DMPI library taken from the developed T-system. Сomparing the effectiveness of the approaches, the performance losses and the prospects for using a hybrid programming style are evaluated. As a result of the conducted experimental studies for different types of computational problems, it was possible to make sure that the efficiency losses are negligible. This allowed to formulate the direction of further work on the T-system and the most promising options for building hybrid applications. Thus, this article presents the results of the comparative tests of LAMMPS application using OpenMPI and using OpenTS DMPI. The test results confirm the effectiveness of the DMPI implementation in the OpenTS parallel programming environment

Legio: fault resiliency for embarrassingly parallel MPI applications

Due to the increasing size of HPC machines, dealing with faults is becoming mandatory due to their high frequency. Natively, MPI cannot handle faults and it stops the execution prematurely when it finds one. With the introduction of ULFM, it is possible to continue the execution, but it requires complex integration with the application. In this paper we propose Legio, a framework that introduces fault resiliency in embarrassingly parallel MPI applications. Legio exposes its features to the application transparently, removing any integration difficulty. After a fault, the execution continues only with the non-failed processes. We also propose a hierarchical alternative, which features lower repair costs on large communicators. We evaluated our solutions on the Marconi100 cluster at CINECA with benchmarks and real-world applications, showing that the overhead introduced by the library is negligible and it does not limit the scalability properties of MPI.