MPI Parallel Research Articles

Enabling rootless Linux Containers in multi-user environments: The udocker tool

Containers are increasingly used as means to distribute and run Linux services and applications. In this paper we describe the architectural design and implementation of udocker, a tool which enables the user to execute Linux containers in user mode. We also present a few practical applications, using a range of scientific codes characterized by different requirements: from single core execution to MPI parallel execution and execution on GPGPUs.

Scalability Evaluation of Cimmino Algorithm for Solving Linear Inequality Systems on Multiprocessors with Distributed Memory

The paper is devoted to a scalability study of Cimmino algorithm for linear inequality systems. This algorithm belongs to the class of iterative projection algorithms. For the analytical analysis of the scalability, the BSF (Bulk Synchronous Farm) parallel computation model is used. An implementation of the Cimmino algorithm in the form of operations on lists using higher-order functions Map and Reduce is presented. An analytical estimation of the upper scalability bound of the algorithm for cluster computing systems is derived. An information about the implementation of Cimmino algorithm on lists in C++ language using the BSF program skeleton and MPI parallel programming library is given. The results of large-scale computational experiments performed on a cluster computing system are demonstrated. A conclusion about the adequacy of the analytical estimations by comparing them with the results of computational experiments is made.

An efficient split-step and implicit pure mesh-free method for the 2D/3D nonlinear Gross–Pitaevskii equations

In this paper, a high-efficient, split-step, and implicit corrected parallel smoothed particle hydrodynamics (SS-ICPSPH) method is developed to simulate the dynamic systems of several nonlinear Schrödinger/Gross–Pitaevskii equations (NLSE/GPE). The proposed method is motivated by the split-step for the equation, the corrected symmetric kernel gradient for the traditional SPH and the implicit scheme for time, respectively. Meanwhile, the MPI parallel technique is introduced to enhance the computational efficiency. Firstly, the numerical accuracy and the merits of the proposed method are tested by solving 2D NLSE, and compared with the analytical results. Secondly, the new method is extended to simulate the 2D/3D two-component GPE, compared with high accuracy finite difference results. Thirdly, the proposed method is extended to investigate the sheet-like vortices in rotating Bose–Einstein condensate. Finally, the implicit corrected SPH scheme is tentatively extended to capture the propagation process of free surface wave in a rectangular pool with initial perturbation. All the numerical results show the ability and the reliability of the proposed method.

Crossing the chasm: how to develop weather and climate models for next generation computers?

Abstract. Weather and climate models are complex pieces of software which include many individual components, each of which is evolving under pressure to exploit advances in computing to enhance some combination of a range of possible improvements (higher spatio-temporal resolution, increased fidelity in terms of resolved processes, more quantification of uncertainty, etc.). However, after many years of a relatively stable computing environment with little choice in processing architecture or programming paradigm (basically X86 processors using MPI for parallelism), the existing menu of processor choices includes significant diversity, and more is on the horizon. This computational diversity, coupled with ever increasing software complexity, leads to the very real possibility that weather and climate modelling will arrive at a chasm which will separate scientific aspiration from our ability to develop and/or rapidly adapt codes to the available hardware. In this paper we review the hardware and software trends which are leading us towards this chasm, before describing current progress in addressing some of the tools which we may be able to use to bridge the chasm. This brief introduction to current tools and plans is followed by a discussion outlining the scientific requirements for quality model codes which have satisfactory performance and portability, while simultaneously supporting productive scientific evolution. We assert that the existing method of incremental model improvements employing small steps which adjust to the changing hardware environment is likely to be inadequate for crossing the chasm between aspiration and hardware at a satisfactory pace, in part because institutions cannot have all the relevant expertise in house. Instead, we outline a methodology based on large community efforts in engineering and standardisation, which will depend on identifying a taxonomy of key activities – perhaps based on existing efforts to develop domain-specific languages, identify common patterns in weather and climate codes, and develop community approaches to commonly needed tools and libraries – and then collaboratively building up those key components. Such a collaborative approach will depend on institutions, projects, and individuals adopting new interdependencies and ways of working.

AFiD-GPU: A versatile Navier–Stokes solver for wall-bounded turbulent flows on GPU clusters

The AFiD code, an open source solver for the incompressible Navier–Stokes equations (http://www.afid.eu), has been ported to GPU clusters to tackle large-scale wall-bounded turbulent flow simulations. The GPU porting has been carried out in CUDA Fortran with the extensive use of kernel loop directives (CUF kernels) in order to have a source code as close as possible to the original CPU version; just a few routines have been manually rewritten. A new transpose scheme has been devised to improve the scaling of the Poisson solver, which is the main bottleneck of incompressible solvers. For large meshes the GPU version of the code shows good strong scaling characteristics, and the wall-clock time per step for the GPU version is an order of magnitude smaller than for the CPU version of the code. Due to the increased performance and efficient use of memory, the GPU version of AFiD can perform simulations in parameter ranges that are unprecedented in thermally-driven wall-bounded turbulence. To verify the accuracy of the code, turbulent Rayleigh–Bénard convection and plane Couette flow are simulated and the results are in excellent agreement with the experimental and computational data that have been published in literature. Program summaryProgram Title: AFiD-GPUProgram Files doi:http://dx.doi.org/10.17632/rwjdg7ry66.1Licensing provisions: MITProgramming language: Fortran 90, CUDA Fortran, MPIExternal routines: PGI, CUDA Toolkit, FFTW3, HDF5Nature of problem: Solving the three-dimensional Navier–Stokes equations coupled with a scalar field in a cubic box bounded between two walls and with periodic boundary conditions in the horizontal directions.Solution method: Second order finite difference method for spatial discretization, third order Runge–Kutta scheme in combination with Crank–Nicolson for the implicit terms for time advancement, two dimensional pencil distributed MPI parallelization, GPU accelerated routines.Additional comments including restrictions and unusual features: The code is available and supported on https://github.com/PhysicsofFluids/AFiD_GPU_opensource.

Scalability of a multi-physics system for forest fire spread prediction in multi-core platforms

Advances in high-performance computing have led to an improvement in modeling multi-physics systems because of the capacity to solve complex numerical systems in a reasonable time. WRF–SFIRE is a multi-physics system that couples the atmospheric model WRF and the forest fire spread model called SFIRE with the objective of considering the atmosphere–fire interactions. In systems like WRF–SFIRE, the trade-off between result accuracy and time required to deliver that result is crucial. So, in this work, we analyze the influence of WRF–SFIRE settings (grid resolutions) into the forecasts accuracy and into the execution times on multi-core platforms using OpenMP and MPI parallel programming paradigms.

A Solution to Image Processing with Parallel MPI I/O and Distributed NVRAM Cache

The paper presents a new approach to parallel image processing using byte addressable, non-volatile memory (NVRAM). We show that our custom built MPI I/O implementation of selected functions that use a distributed cache that incorporates NVRAMs located in cluster nodes can be used for efficient processing of large images. We demonstrate performance benefits of such a solution compared to a traditional implementation without NVRAM for various sizes of buffers used to read image parts, process and write back to storage. We also show that our implementation benefits from overlapping reading subsequent images while processing already loaded ones. We present results obtained in a cluster environment for three parallel implementation of blur, multipass blur and Sobel filters, for various NVRAM parameters such as latencies and bandwidth values.

Wide Angular Sweeping of Dynamic Electromagnetic Responses From Large Targets by MPI Parallel Skeletonization

It has been revealed that the interpolative decomposition (ID) skeletonization can greatly accelerate the wide angular sweeping, which involves repeatedly solving many/massive incidents. However, the peak memory usage required by skeletonization may be a bottleneck of that algorithm. In this communication, a message passing interface parallel skeletonization scheme is developed to alleviate the memory usage and to incorporate with the parallel multilevel fast multipole algorithm (MLFMA) for large targets, which are hundreds of wavelengths in size. The parallel scheme includes two ingredients. One is to feed into the ID, a size-reduced matrix to find skeleton right-hand sides. This matrix is produced upon skeleton RWGs instead of their original counterpart. The other is to integrate the parallel ID into the parallel MLFMA to alleviate the peak memory usage of skeletonization on each participating process. Numerical experiments show that the peak memory usage for the skeletonization can be less than 2 GB in each process for the target about 300 wavelengths in size.

CoFFEE: Corrections For Formation Energy and Eigenvalues for charged defect simulations

Charged point defects in materials are widely studied using Density Functional Theory (DFT) packages with periodic boundary conditions. The formation energy and defect level computed from these simulations need to be corrected to remove the contributions from the spurious long-range interaction between the defect and its periodic images. To this effect, the CoFFEE code implements the Freysoldt–Neugebauer–Van de Walle (FNV) correction scheme. The corrections can be applied to charged defects in a complete range of material shapes and size: bulk, slab (or two-dimensional), wires and nanoribbons. The code is written in Python and features MPI parallelization and optimizations using the Cython package for slow steps. Program summaryProgram title: CoFFEEProgram Files doi:http://dx.doi.org/10.17632/s2x4d542dc.1Licensing provisions: BSD 3-clauseProgramming language: PythonExternal routines/libraries: numpy, scipy, mpi4py, matplotlibNature of problem: Most electronic structure codes based on Density Functional Theory use periodic boundary conditions. This leads to spurious electrostatic interactions during simulation of charged defects, which affects the computed defect formation energy and the defect eigenvalue.Solution method: We implement the Freysoldt–Neugebauer–Van de Walle (FNV) correction scheme to correct the defect formation energy and eigenvalues. Our implementation can be applied to charged defects in 3D bulk materials as well as materials having 2D and 1D geometries.

A Parallelization Research for FY Satellite Rainfall Estimate Day Knock off Product Algorithm

With the development of satellite remote sensing technology, more and more requirements are put forward on the timeliness and stability of the satellite weather service system. The FY satellite rainfall estimate day knock off product algorithm runs longer, about 20 minutes, which affects the estimated rainfall product generated timeliness. Research and development of parallel optimization algorithms based on the needs of satellite meteorological services and their effectiveness in practical applications are necessary ways to enhance the high-performance and high-availability capabilities of satellite meteorological services. So aiming at this problem, we started the parallel algorithm research based on the analysis of precipitation estimation algorithm. Firstly, we explained the steps of precipitation estimated date knock off product algorithm; secondly, we analyzed the four main calculation module calculating the amount of algorithms; thirdly, multithreaded parallel algorithm and MPI parallelization was designed. Finally, the multithreaded parallel and MPI parallelization were realized. Experimental results show that the multithreaded parallel and MPI parallelization algorithm could greatly improve the overall degree of computational efficiency. And, MPI parallelization mode has a higher operating efficiency. The performance of parallel processing is closely related to the architecture of the computer. From the perspective of service scheduling and product algorithms, the MPI parallelization approach is adopted to achieve the purpose of improving service quality.

MPI Parallel Monte Carlo Framework for the Reliability Analysis of Highway Bridges

AbstractCurrent emphasis on evaluating the resilience of civil communities to various natural and man-made hazards has required expanding the scale of structural analysis problems from performing d...

On efficiency of fire simulation realization: parallelization with greater number of computational meshes

Current fire simulation systems are capable to utilize advantages of high-performance computer (HPC) platforms available and to model fires efficiently in parallel. In this paper, efficiency of a corridor fire simulation on a HPC computer cluster is discussed. The parallel MPI version of Fire Dynamics Simulator is used for testing efficiency of selected strategies of allocation of computational resources of the cluster using a greater number of computational cores. Simulation results indicate that if the number of cores used is not equal to a multiple of the total number of cluster node cores there are allocation strategies which provide more efficient calculations.

An efficient and accurate OpenMP-MPI based 3D beam tracing method for radio propagation simulation in large scale urban environments

A modified adaptive division beam tracing (MADBT) technique based on the localized kd-tree and the OpenMP–MPI parallel techniques is presented for the simulation of radio propagation in large-scale urban environments. Beams launched from a transmitter are recursively traced by applying the MADBT method and beam trees recorded the propagation path info are generated. By traversing the beam trees, all the valid propagation (transmitted, reflected and diffracted) paths are efficiently determined. Then the receiver power strength can be calculated based on geometrical optics (GO) and uniform theory of diffraction along the arrival paths. By tracing the continuous beams instead of a number of rays, the proposed method performs higher efficiency than the widely used ray tracing method. During the beam tracing procedure, several techniques are adopted to accelerate the calculation. Firstly, a localized kd-tree is built according to the coverage area of the transmitter, which is more efficient than the global kd-tree to do the beam-triangle intersection during the tracing procedure. Secondly, a virtual simulation face is generated to speed up the searching for ray paths. Lastly, the OpenMP and the MPI parallel technique are implemented to further enhance efficiency of the proposed method. As expected, the simulated results for the realistic Malaysia urban environment show good agreement with measured results and performance high efficiency.

Quantitative 3D phase field modelling of solidification using next-generation adaptive mesh refinement

Phase field (PF) models are one of the most popular methods for simulating solidification microstructures due to their fundamental connections to the physics of phase transformations. However, these methods are numerically very stiff due to the multiple length scales in a solidifying material, from the nanoscopic solid-liquid interface, to dendritic structures on the order of hundreds of microns. While this problem can be greatly alleviated by thin-interface analytical treatments of the PF equations, additional numerical methods are required to explore experimentally relevant sample sizes and times scales. It was shown about 18years ago that the use of dynamic adaptive mesh refinement (AMR) can alleviate this problem by exploiting the simple fact that the majority of the solidification kinetics occur at the solid-liquid interface, which scales with a lower dimensionality than the embedding system itself. AMR methods, together with asymptotic analysis, nowadays provide one of the most efficient numerical strategies for self-consistent quantitative PF modelling of solidification microstructure processes. This paper highlights the latest developments in the AMR technique for 3D modelling of solidification using classical phase field equations. This includes a move away from finite element techniques to faster finite differencing through the use of dynamic mini-meshes which are each associated with each node of a 3D Octree data structure, and distributed MPI parallelism that uses a new communication algorithm to decompose a 3D domain into multiple adaptive meshes that are spawned on separate cores. The numerical technique is discussed, followed by demonstrations of the new AMR algorithm on select benchmark solidification problems, as well as some illustrations of multi-phase modelling using a recently developed multi-order parameter phase field model.

Improvement of workload balancing using parallel loop self-scheduling on Intel Xeon Phi

In recent years, Intel promotes its new product Xeon Phi coprocessor, which is similar to the x86 architecture coprocessor. It has about 60 cores and can be regarded as a single computing node, with the computing power that cannot be ignored. This work aims to improve the workload balance by parallel loop self-scheduling scheme performed on Xeon Phi-based computer cluster. The proposed concept is implemented by hybrid MPI and OpenMP parallel programming in C language. Since parallel loop self-scheduling composes of static and dynamic allocation, weighting algorithm is adopted in the static part, while the well-known loop self-scheduling is adopted in dynamic part. The loop block is partitioned according to the weighting of MIC and HOST nodes. Accordingly, Xeon Phi with many-core is adopted to implement parallel loop self-scheduling. Finally, we test the performance in the experiments by four applicable problems: matrix multiplication, sparse matrix multiplication, Mandelbrot set and circuit meet. The experimental results indicate how to do the weight allocation and which scheduling method can achieve the best performance.

BeatBox-HPC simulation environment for biophysically and anatomically realistic cardiac electrophysiology.

The BeatBox simulation environment combines flexible script language user interface with the robust computational tools, in order to setup cardiac electrophysiology in-silico experiments without re-coding at low-level, so that cell excitation, tissue/anatomy models, stimulation protocols may be included into a BeatBox script, and simulation run either sequentially or in parallel (MPI) without re-compilation. BeatBox is a free software written in C language to be run on a Unix-based platform. It provides the whole spectrum of multi scale tissue modelling from 0-dimensional individual cell simulation, 1-dimensional fibre, 2-dimensional sheet and 3-dimensional slab of tissue, up to anatomically realistic whole heart simulations, with run time measurements including cardiac re-entry tip/filament tracing, ECG, local/global samples of any variables, etc. BeatBox solvers, cell, and tissue/anatomy models repositories are extended via robust and flexible interfaces, thus providing an open framework for new developments in the field. In this paper we give an overview of the BeatBox current state, together with a description of the main computational methods and MPI parallelisation approaches.

VCC: A framework for building containerized reproducible cluster software environments

The problem of portability and reproducibility of the software used to conduct computational experiments has recently come to the fore. Container virtualisation has proved to be a powerful tool to achieve portability of a code and it's execution environment, through runtimes such as Docker, LXC, Singularity and others - without the performance cost of traditional Virtual Machines (Chamberlain, Invenshure, and Schommer 2014; Felter et al. 2014). However, scientific software often depends on a system foundation that provides middleware, libraries, and other supporting software in order for the code to execute as intended. Typically, container virtualisation addresses only the portability of the code itself, which does not make it inherently reproducible. For example, a containerized MPI application may offer binary compatibility between different systems, but for execution as intended, it must be run on an existing cluster that provides the correct interfaces for parallel MPI execution. As a greater demand to accomodate a diverse range of disciplines is placed on high performance and cluster resources, the ability to quickly create and teardown reproducible, transitory virtual environments that are tailored for an individual task or experiment will be essential. The Virtual Container Cluster (VCC) is a framework for building containers that achieve this goal, by encapsulating a parallel application along with an execution model, through a set of dependency linked services and built-in process orchestration. This promotes a high degree of portability, and offers easier reproducibility by shipping the application along with the foundation required to execute it - whether that be an MPI cluster, big data processing framework, bioinformatics pipeline, or any other execution model (Higgins, Holmes, and Venters 2017).

Monteswitch : A package for evaluating solid–solid free energy differences via lattice-switch Monte Carlo

Lattice-switch Monte Carlo (LSMC) is a method for evaluating the free energy between two given solid phases. LSMC is a general method, being applicable to a wide range of problems and interatomic potentials. Furthermore it is extremely efficient, ostensibly more efficient than other existing general methods. Here we introduce a package, monteswitch, which can be used to perform LSMC simulations. The package can be used to evaluate the free energy differences between pairs of solid phases, including multicomponent phases, via LSMC for atomic (i.e., non-molecular) systems in the NVT and NPT ensembles. It could also be used to evaluate the free energy cost associated with interfaces and defects. Regarding interatomic potentials, monteswitch currently supports various commonly-used pair potentials, including the hard-sphere, Lennard-Jones, and Morse potentials, as well as the embedded atom model. However the main strength of the package is its versatility: it is designed so that users can easily implement their own potentials.Program summaryProgram Title: monteswitchProgram Files doi:http://dx.doi.org/10.17632/zzy7jh9ynk.1Licensing provisions: MITProgramming language: Fortran 95, MPINature of problem: Calculating the free energy difference between two solid systems.Solution method: Lattice-switch Monte Carlo (LSMC) [1] is a versatile and efficient method for evaluating the free energy difference between two solid phases. The package presented here allows LSMC simulations to be performed for a variety of interatomic potentials, including commonly-used pair potentials and the embedded atom model. Furthermore the package is designed so that users can easily implement their own potentials. The package supports LSMC simulations in the NVT and NPT ensembles, and can treat multicomponent systems. A version of the main program is included which is parallelised using MPI. This program parallelises the LSMC calculation by simulating multiple replicas of the system in parallel.External routines/libraries: To perform parallel simulations MPI is required (but MPI is not required to perform serial simulations).Restrictions: monteswitch cannot treat molecular systems, i.e., systems in which the particles exhibit rotational degrees of freedom, and is restricted to systems which can be represented within an orthorhombic supercell. Furthermore, the interatomic potential is ‘hard-coded’ in the sense that implementing a different potential requires that the package be recompiled.Additional comments: monteswitch includes programs to assist with the creation of input files and the post-processing of output files created by the main Monte Carlo programs. A user manual, a suite of test cases, a worked example, and a collection of plug-ins to implement various commonly-used interatomic potentials are also included with the package. [1]A.D. Bruce, N.B. Wilding & G.J. Ackland, Phys. Rev. Lett. 79 3002 (1997)

High performance shallow water kernels for parallel overland flow simulations based on FullSWOF2D

We describe code optimization and parallelization procedures applied to the sequential overland flow solver FullSWOF2D. Major difficulties when simulating overland flows comprise dealing with high resolution datasets of large scale areas which either cannot be computed on a single node either due to limited amount of memory or due to too many (time step) iterations resulting from the CFL condition. We address these issues in terms of two major contributions. First, we demonstrate a generic step-by-step transformation of the second order finite volume scheme in FullSWOF2D towards MPI parallelization. Second, the computational kernels are optimized by the use of templates and a portable vectorization approach. We discuss the load imbalance of the flux computation due to dry and wet cells and propose a solution using an efficient cell counting approach. Finally, scalability results are shown for different test scenarios along with a flood simulation benchmark using the Shaheen II supercomputer.

Multi-agent large-scale parallel crowd simulation

This paper presents design, implementation and performance results of a new modular, parallel, agent-based and large scale crowd simulation environment. A parallel application, implemented with C and MPI, was implemented and run in this parallel environment for simulation and visualization of an evacuation scenario at Gdansk University of Technology, Poland and further in the area of districts of Gdansk. The application uses a parallel MPI I/O run on two different clusters and a two or three node Parallel File System (PFS) to store a current state in a file. In order to make this implementation efficient, we used our previously developed and tuned Byte-addressable Non-volatile RAM Distributed Cache - a solution that allows to access small data chunks from spread locations within a file efficiently. We have presented application execution times versus the number of agents (up to 100000), versus the number of simulation iterations (up to 25000), versus map size (up to 6km2) and versus the number of processes (up to more than 650) showing high speed-ups.