Distributed Memory Supercomputers Research Articles

ParVoro++: A scalable parallel algorithm for constructing 3D Voronoi tessellations based on kd-tree decomposition

The Voronoi tessellation is a fundamental geometric data structure which has numerous applications in various scientific and technological fields. For large particle datasets, computing Voronoi tessellations must be conducted in parallel on a distributed-memory supercomputer in order to satisfy time and memory-size constraints. However, due to load balance and communication, the parallelization of the Voronoi tessellation renders a challenge. In this paper, we present a scalable parallel algorithm for constructing 3D Voronoi tessellations, which evenly distributes the input particles between blocks through kd-tree decomposition. In order to construct the correct global Voronoi topology, we investigate both parametric and non-parametric methods for particle communication among the blocks of a spatial decomposition. The algorithm is implemented exploiting process-level and thread-level parallelization and can be used in a diverse architectural landscape. Using datasets containing up to 330 million particles, we show that our algorithm achieves parallel efficiency up to 57% using 4096 cores on a distributed-memory computer. Moreover, we compare our algorithm with previous attempts to parallelize Voronoi tessellations showing encouraging improvements in terms of computation time.

GenASiSBasics: Object-oriented utilitarian functionality for large-scale physics simulations (Version 4)

GenASiSBasics: Object-oriented utilitarian functionality for large-scale physics simulations (Version 4)

Improving ocean modeling software NEMO 4.0 benchmarking and communication efficiency

Abstract. Communications in distributed memory supercomputers are still limiting scalability of geophysical models. Considering the recent trends of the semiconductor industry, we think this problem is here to stay. We present the optimizations that have been implemented in the 4.0 version of the ocean model NEMO to improve its scalability. Thanks to the collaboration of oceanographers and HPC experts, we identified and removed the unnecessary communications in two bottleneck routines, the computation of free surface pressure gradient, and the forcing in the straight or unstructured open boundaries. Since a wrong parallel decomposition choice could undermine computing performance, we impose its automatic definition in all cases, including when subdomains containing land points only are excluded from the decomposition. For a smaller audience of developers and vendors, we propose a new benchmark configuration, which is easy to use while offering the full complexity of operational versions.

High order finite volume schemes with IMEX time stepping for the Boltzmann model on unstructured meshes

In this work, we present a family of time and space high order finite volume schemes for the solution of the full Boltzmann equation. The velocity space is approximated by using a discrete ordinate approach while the collisional integral is approximated by spectral methods. The space reconstruction is implemented by integrating the distribution function, which describes the state of the system, over arbitrarily shaped and closed control volumes using a Central Weighted ENO (CWENO) technique. Compared to other reconstruction methods, this approach permits to keep compact stencil sizes which is a remarkable property in the context of kinetic equations due to the considerable demand of computational resources. The full discretization is then obtained by combining the previous phase-space approximation with high order Implicit–Explicit (IMEX) Runge–Kutta schemes. These methods guarantee stability, accuracy and preservation of the asymptotic state. Comparisons of the Boltzmann model with simpler relaxation type kinetic models (like BGK) are proposed showing the capability of the Boltzmann equation to capture different physical solutions. The theoretical order of convergence is numerically measured in different regimes and the methods are tested on several standard two-dimensional benchmark problems in comparison with Direct Simulation Monte Carlo results. The article ends with a prototype engineering problem consisting of a subsonic and a supersonic flow around a NACA 0012 airfoil. All test cases are run with MPI parallelization on several cores, thus making the proposed methods suitable for parallel distributed memory supercomputers.

A Massively Parallel Hybrid Finite Volume/Finite Element Scheme for Computational Fluid Dynamics

In this paper, we propose a novel family of semi-implicit hybrid finite volume/finite element schemes for computational fluid dynamics (CFD), in particular for the approximate solution of the incompressible and compressible Navier-Stokes equations, as well as for the shallow water equations on staggered unstructured meshes in two and three space dimensions. The key features of the method are the use of an edge-based/face-based staggered dual mesh for the discretization of the nonlinear convective terms at the aid of explicit high resolution Godunov-type finite volume schemes, while pressure terms are discretized implicitly using classical continuous Lagrange finite elements on the primal simplex mesh. The resulting pressure system is symmetric positive definite and can thus be very efficiently solved at the aid of classical Krylov subspace methods, such as a matrix-free conjugate gradient method. For the compressible Navier-Stokes equations, the schemes are by construction asymptotic preserving in the low Mach number limit of the equations, hence a consistent hybrid FV/FE method for the incompressible equations is retrieved. All parts of the algorithm can be efficiently parallelized, i.e., the explicit finite volume step as well as the matrix-vector product in the implicit pressure solver. Concerning parallel implementation, we employ the Message-Passing Interface (MPI) standard in combination with spatial domain decomposition based on the free software package METIS. To show the versatility of the proposed schemes, we present a wide range of applications, starting from environmental and geophysical flows, such as dambreak problems and natural convection, over direct numerical simulations of turbulent incompressible flows to high Mach number compressible flows with shock waves. An excellent agreement with exact analytical, numerical or experimental reference solutions is achieved in all cases. Most of the simulations are run with millions of degrees of freedom on thousands of CPU cores. We show strong scaling results for the hybrid FV/FE scheme applied to the 3D incompressible Navier-Stokes equations, using millions of degrees of freedom and up to 4096 CPU cores. The largest simulation shown in this paper is the well-known 3D Taylor-Green vortex benchmark run on 671 million tetrahedral elements on 32,768 CPU cores, showing clearly the suitability of the presented algorithm for the solution of large CFD problems on modern massively parallel distributed memory supercomputers.

High Performance Computing Framework for Tera-Scale Database Search of Mass Spectrometry Data.

Database peptide search algorithms deduce peptides from mass spectrometry (MS) data. There has been substantial effort in improving their computational efficiency to achieve larger and more complex systems biology studies. However, modern serial and high-performance computing (HPC) algorithms exhibit sub-optimal performance mainly due to their ineffective parallel designs (low resource utilization), and high overhead costs. We present an HPC framework, called HiCOPS, for efficient acceleration of the database peptide search algorithms on distributed-memory supercomputers. HiCOPS provides, on average, more than 10-fold improvement in speed, and superior parallel performance over several existing HPC database search software. We also formulate a mathematical model for performance analysis and optimization, and report near-optimal results for several key metrics including strong-scale efficiency, hardware utilization, load-balance, inter-process communication and I/O overheads. The core parallel design, techniques, and optimizations presented in HiCOPS are search-algorithm independent and can be extended to efficiently accelerate the existing and future algorithms and software.

MIN3P-HPC: A High-Performance Unstructured Grid Code for Subsurface Flow and Reactive Transport Simulation

The numerical simulation of flow and reactive transport in porous media with complex domains is nontrivial. This paper presents a method to implement fully unstructured grid capabilities into the well-established software ParMIN3P-THCm, a process-based numerical model designed for the investigation of subsurface fluid flow and multicomponent reactive transport in variably saturated porous media with parallelization capability. The enhanced code, MIN3P-HPC, is modularized to support different cell types, spatial discretization methods and gradient reconstruction methods. MIN3P-HPC uses a vertex-centered control volume method with consideration of both vertex-based and cell-based material properties (e.g., permeability). A flexible parallelization scheme based on domain decomposition and thread acceleration was implemented, which allows the use of OpenMP, MPI and hybrid MPI-OpenMP, making optimized use of computer resources ranging from desktop PCs to distributed memory supercomputers. The code was verified by comparing the results obtained with the unstructured grid version to those produced by the structured grid version. Numerical accuracy was also verified against analytical solutions for 2D and 3D solute transport, and by comparison with third-party software using different cell types. Parallel efficiency of OpenMP, MPI and hybrid MPI-OpenMP versions was examined through a series of solute transport and reactive transport test cases. The results demonstrate the versatility and enhanced performance of MIN3P-HPC for reactive transport simulation.

High order central WENO-Implicit-Explicit Runge Kutta schemes for the BGK model on general polygonal meshes

In this work, a family of high order accurate Central Weighted ENO (CWENO) finite volume schemes for the solution of nonlinear kinetic equation of relaxation type is presented. After discretization of the velocity space by using a discrete ordinate approach, the space reconstruction is realized by integration over conformal arbitrary shaped closed space control volumes in a CWENO fashion. Compared to other WENO methods on unstructured meshes, in the method here presented, the total stencil size is the minimum possible and the linear weights can be arbitrarily chosen. These two aspects make their use for kinetic equations and the practical implementation on general unstructured meshes particularly interesting. The full discretization is then obtained by combining the previous phase-space approximation with an Implicit-Explicit Runge Kutta high order time discretization which guarantees stability, accuracy and preservation of the asymptotic state. In particular, to guarantee in the finite volume framework space accuracy higher than two, a new class of IMEX methods has been set into place and its properties have been studied. The formal order of accuracy is numerically measured for different regimes, computational performances of the proposed class of methods are tested on several standard two dimensional benchmark problems for kinetic equations. The novel methods are finally applied to a prototype engineering problem consisting in a supersonic flow around a NACA 0012 airfoil. In our computations we employ up to ≈325 millions of degrees of freedom and 256 GB of RAM run on 128 cores with Fortran-MPI providing evidence that the above schemes are suitable for implementation on parallel distributed memory supercomputers.

Understanding repulsively mediated superconductivity of correlated electrons via massively parallel density matrix renormalization group

The so-called minimal models of unconventional superconductivity are lattice models of interacting electrons derived from materials in which electron pairing arises from purely repulsive interactions. Showing unambiguously that a minimal model actually can have a superconducting ground state remains a challenge at nonperturbative interactions. We make a significant step in this direction by computing ground states of the 2D \mbox{U-V} Hubbard model - the minimal model of the quasi-1D superconductors - by parallelized DMRG, which allows for systematic control of any bias and that is sign-problem-free. Using distributed-memory supercomputers and leveraging the advantages of the \mbox{U-V} model, we can treat unprecedented sizes of 2D strips and extrapolate their spin gap both to zero approximation error and the thermodynamic limit. Our results for the spin gap are shown to be compatible with a spin excitation spectrum that is either fully gapped or has zeros only in discrete points, and conversely that a Fermi liquid or magnetically ordered ground state is incompatible with them. Coupled with the enhancement to short-range correlations that we find exclusively in the $d_{xy}$ pairing-channel, this allows us to build an indirect case for the ground state of this model having superconducting order in the full 2D limit, and ruling out the other main possible phases, magnetic orders and Fermi liquids.

GenASiSBasics: Object-oriented utilitarian functionality for large-scale physics simulations (Version 3)

GenASiS Basics provides Fortran 2003 classes furnishing extensible object-oriented utilitarian functionality for large-scale physics simulations on distributed memory supercomputers. This functionality includes physical units and constants; display to the screen or standard output device; message passing; I/O to disk; and runtime parameter management and usage statistics. This revision---Version 3 of Basics---includes a significant name change, some minor additions to functionality, and a major addition to functionality: infrastructure facilitating the offloading of computational kernels to devices such as GPUs.

TIGAr: Automating isogeometric analysis with FEniCS

This paper introduces tIGAr, a library for using the open-source finite element (FE) automation software FEniCS to perform isogeometric analysis (IGA). The library uses a global variant of Bézier extraction to avoid modifying the finite element assembly procedures of FEniCS. This makes much of the convenient functionality of FEniCS directly available for IGA. General rational splines are implemented through an abstraction that sees only an extracted representation of the IGA function space. Through this abstraction, an enormous variety of spline spaces can be used for analysis, so long as a corresponding preprocessor is developed for each one, implementing a simple interface. As examples, we discuss preprocessors for B-splines specified analytically, non-uniform rational B-splines (NURBS) designed using the open-source software igakit, and T-splines designed using commercial software. We then demonstrate the implementation of solvers for several non-trivial partial differential equations that benefit from IGA. We also evaluate the parallel performance of tIGAr on a distributed-memory supercomputer. Finally, we outline possibilities for further development of IGA in FEniCS. Source code for tIGAr is continuously updated online at https://github.com/david-kamensky/tIGAr.

Tinker 8: Software Tools for Molecular Design.

The Tinker software, currently released as version 8, is a modular molecular mechanics and dynamics package written primarily in a standard, easily portable dialect of Fortran 95 with OpenMP extensions. It supports a wide variety of force fields, including polarizable models such as the Atomic Multipole Optimized Energetics for Biomolecular Applications (AMOEBA) force field. The package runs on Linux, macOS, and Windows systems. In addition to canonical Tinker, there are branches, Tinker-HP and Tinker-OpenMM, designed for use on message passing interface (MPI) parallel distributed memory supercomputers and state-of-the-art graphical processing units (GPUs), respectively. The Tinker suite also includes a tightly integrated Java-based graphical user interface called Force Field Explorer (FFE), which provides molecular visualization capabilities as well as the ability to launch and control Tinker calculations.

GenASiS Basics: Object-oriented utilitarian functionality for large-scale physics simulations (Version 2)

GenASiSBasics provides Fortran 2003 classes furnishing extensible object-oriented utilitarian functionality for large-scale physics simulations on distributed memory supercomputers. This functionality includes physical units and constants; display to the screen or standard output device; message passing; I/O to disk; and runtime parameter management and usage statistics. This revision–Version 2 of Basics–makes mostly minor additions to functionality and includes some simplifying name changes. New version program summaryProgram Title: SineWaveAdvection, SawtoothWaveAdvection, and RiemannProblem (fluid dynamics example problems illustrating GenASiSBasics); ArgonEquilibrium and ClusterFormation (molecular dynamics example problems illustrating GenASiSBasics)Program Files doi:http://dx.doi.org/10.17632/6w9ygpygmc.1Licensing provisions: GPLv3Programming language: Fortran 2003 (tested with gfortran 6.1.0, Intel Fortran 16.0.3, Cray Compiler 8.5.3)Journal reference of previous version: Computer Physics Communications, 196 (2015) 506Does the new version supersede the previous version?: YesReasons for the new version: This version makes mostly minor additions to functionality and includes some simplifying name changes.Summary of revisions: Several additions to functionality are minor. Two new singleton objects are KIND_SMALL and KIND_TINY, for smaller sized numbers than those specified by the previously available KIND_DEFAULT. The class MeasuredValueForm can now handle some more complicated cases of unit string processing. The numerical values in the CONSTANT singleton have been updated to 2016 values [3], and CONSTANT and UNIT contain a few additional members.A new class TimerForm can be used to track the wall time occupied by various segments of code. The PROGRAM_HEADER singleton now contains an array member of this new class. With calls like [Display omitted] the user can initialize their own timers; on return iMyTimer contains the index of the newly initialized timer. The calls [Display omitted] and [Display omitted] should surround the block of code to be timed. The information displayed by calling the ShowStatistics method of PROGRAM_HEADER includes data from all initialized timers, including one for overall execution time which is present by default.The code now expects to be compiled with OpenMP, typically by applying compiler flags. Strictly speaking this is only required for the PROGRAM_HEADER singleton, which queries the number of threads via a library call. In GenASiSBasics, OpenMP directives (which appear as comments as far as the Fortran 2003 standard is concerned) are only used in the Clear and Copy commands.There have been a number of name changes, mostly for simplification and consistency. These include the classes in ArrayArrays, where for example ArrayInteger_1D_Form is now simply Integer_1D_Form. Similar streamlining changes have been made to MessagePassing classes: IncomingMessageArrayRealForm is now MessageIncoming_1D_R_Form, for instance. The class VariableGroupArrayMetadata is now VariableGroup_1D_Form. The name ParametersStreamForm has been changed by one character (deletion of an s) to ParameterStreamForm. The member Selected of VariableGroupForm has been changed to iaSelected, where the prefix ia is a conventional prefix we use for an array of array indices.The interface and functionality of the SetGrid member of StructuredGridImageForm have been modified so as not to include boundary cells exterior to the computational domain, which prevented display of the computational domain in 3D plots with VisIt [4] unless a Box operator was applied. See the fluid dynamics examples for the modified usage.Finally, version 4.10 of the Silo library [2] introduced an include file named silo_f9x.inc, which the FileSystem classes of GenASiSBasics now expect to be available instead of silo.inc.Nature of problem: By way of illustrating GenASiSBasics functionality, solve example fluid dynamics and molecular dynamics problems.Solution method: For fluid dynamics examples, finite-volume. For molecular dynamics examples, leapfrog and velocity-Verlet integration.External routines/libraries: MPI [1] and Silo [2]Additional comments including Restrictions and Unusual features:The example problems named above are not ends in themselves, but serve to illustrate our object-oriented approach and the functionality available though GenASiSBasics. In addition to these more substantial examples, we provide individual unit test programs for each of the classes comprised by GenASiSBasics.GenASiSBasics is available in the CPC Program Library and also at https://github.com/GenASiS. [1]http://www.mcs.anl.gov/mpi/[2]https://wci.llnl.gov/simulation/computer-codes/silo[3]C. Patrignani et al. (Particle Data Group), Chin. Phys. C 40 (2016) 100001[4]https://wci.llnl.gov/simulation/computer-codes/visit

Central Weighted ENO Schemes for Hyperbolic Conservation Laws on Fixed and Moving Unstructured Meshes

We present a novel arbitrary high order accurate central WENO spatial reconstruction procedure (CWENO) for the solution of nonlinear systems of hyperbolic conservation laws on fixed and moving unstructured simplex meshes in two and three space dimensions. Starting from the given cell averages of a function on a triangular or tetrahedral control volume and its neighbors, the nonlinear CWENO reconstruction yields a high order accurate and essentially non-oscillatory polynomial that is defined everywhere in the cell. Compared to other WENO schemes on unstructured meshes, the total stencil size is the minimum possible one, as in classical point-wise WENO schemes of Jiang and Shu. However, the linear weights can be chosen arbitrarily, which makes the practical implementation on general unstructured meshes particularly simple. We make use of the piecewise polynomials generated by the CWENO reconstruction operator inside the framework of fully discrete and high order accurate one-step ADER finite volume schemes on fixed Eulerian grids as well as on moving Arbitrary-Lagrangian-Eulerian (ALE) meshes. The computational efficiency of the high order finite volume schemes based on the new CWENO reconstruction is tested on several two- and three-dimensional systems of hyperbolic conservation laws and is found to be more efficient in terms of memory consumption and computational efficiency with respect to classical WENO reconstruction schemes on unstructured meshes. We also provide evidence that the new algorithm is suitable for implementation on massively parallel distributed memory supercomputers, showing two numerical examples run with more than one billion degrees of freedom in space. To our knowledge, at present these are the largest simulations ever run with unstructured WENO finite volume schemes.

A Parallel Approach for the Generation of Unstructured Meshes with Billions of Elements on Distributed-Memory Supercomputers

This paper describes a parallel approach for the rapid generation of ultra-large-scale unstructured meshes on distributed-memory supercomputers. A medium-sized initial mesh is prepared first. Afterwards, a two-level domain decomposition (DD) strategy is used to split and distribute the initial mesh to different cores. Finally, the parallel mesh generation, comprising a recursive procedure which includes parallel surface recovery, parallel boundary updating, and parallel mesh multiplication, is performed. The two-level DD differentiates the intra-node and inter-node communication to reduce communication overheads. A global indexing and updating scheme is used to make the mesh multiplication devoid of communication. A new parallel surface recovery algorithm without communication is developed to maintain the fidelity of the resulting mesh model to the original geometric model. Tests of the parallel approach for some real-life problems on supercomputers (Dawning-5000A and Tianhe-2) are presented. Issues regarding the speedup, parallel efficiency, and mesh quality are discussed. Results show that the proposed parallel approach has a reasonably good scalability, that the quality of the resulting mesh is improved, and that ultra-large-scale meshes with billions of elements can be generated quickly.

GenASiS Basics: Object-oriented utilitarian functionality for large-scale physics simulations

Aside from numerical algorithms and problem setup, large-scale physics simulations on distributed-memory supercomputers require more basic utilitarian functionality, such as physical units and constants; display to the screen or standard output device; message passing; I/O to disk; and runtime parameter management and usage statistics. Here we describe and make available Fortran 2003 classes furnishing extensible object-oriented implementations of this sort of rudimentary functionality, along with individual ‘unit test’ programs and larger example problems demonstrating their use. These classes compose the Basics division of our developing astrophysics simulation code GenASiS (General Astrophysical Simulation System), but their fundamental nature makes them useful for physics simulations in many fields. Program summaryProgram title:GenASiSCatalogue identifier: AEXE_v1_0Program summary URL:http://cpc.cs.qub.ac.uk/summaries/AEXE_v1_0.htmlProgram obtainable from: CPC Program Library, Queen’s University, Belfast, N. IrelandLicensing provisions: Creative Commons Attribution - Non Commercial - ShareALike 4.0 InternationalNo. of lines in distributed program, including test data, etc.: 32863No. of bytes in distributed program, including test data, etc.: 148873Distribution format: tar.gzProgramming language: Fortran 2003 (tested with gfortran 4.9.2, Intel Fortran 15, NAG Fortan 5.3.1, Cray Compiler 8.2.5).Computer: PC, cluster, supercomputer.Operating system: Linux, Unix.RAM: For example problems, depends on user-specified problem size and number of processes. The fluid dynamics problems with 1283 cells on 8 processes use about 300 MB per process. The molecular dynamics problems with 6912 particles on 12 processes use about 20 MB per process.Classification: 4.14, 6.5, 20.External routines: MPI [1] and Silo [2]Nature of problem: By way of illustrating GenASiSBasics functionality, solve example fluid dynamics and molecular dynamics problems.Solution method: For fluid dynamics examples, finite-volume. For molecular dynamics examples, leapfrog and velocity-Verlet integration.Unusual features: The example problems named above are not ends in themselves, but serve to illustrate our object-oriented approach and the functionality available though GenASiSBasics. In addition to these more substantial examples, we provide individual unit test programs for each of the classes comprised by GenASiSBasics.Additional comments: A version of the GenASiSBasics source code is available from the CPC program library with this publication, and minor revisions will be maintained at http://astro.phys.utk.edu/activities:genasis.Running time: For example problems, depends on user-specified problem size and number of processes. The fluid dynamics problems with 1283 cells on 8 processes take about ten minutes of wall clock time on a Cray XC30. The molecular dynamics problems with 6912 particles for 10000 time steps on 12 processes take a little over an hour on a Cray XC30.

A work stealing based approach for enabling scalable optimal sequence homology detection

Sequence homology detection is central to a number of bioinformatics applications including genome sequencing and protein family characterization. Given millions of sequences, the goal is to identify all pairs of sequences that are highly similar (or “homologous”) on the basis of alignment criteria. While there are optimal alignment algorithms to compute pairwise homology, their deployment for large-scale is currently not feasible; instead, heuristic methods are used at the expense of quality. Here, we present the design and evaluation of a parallel implementation for conducting optimal homology detection on distributed memory supercomputers. Our approach uses a combination of techniques from asynchronous load balancing (viz. work stealing, dynamic task counters), data replication, and exact-matching filters to achieve homology detection at scale. Results for 2.56 M sequences on up to 8K cores show parallel efficiencies of ∼75%–100%, a time-to-solution of 33 s, and a rate of ∼2.0M alignments per second.

Genome Assembly Framework on Massively Parallel, Distributed Memory Supercomputers

Article Genome Assembly Framework on Massively Parallel, Distributed Memory Supercomputers was published on September 7, 2013 in the journal Biomedical Engineering / Biomedizinische Technik (volume 58, issue SI-1-Track-M).

Parallel Dichotomy Algorithm for solving tridiagonal system of linear equations with multiple right-hand sides

A parallel algorithm for solving a series of matrix equations with a constant tridiagonal matrix and different right-hand sides is proposed and studied. The process of solving the problem is represented in two steps. The first preliminary step is calculating some rows of the inverse matrix of system of linear algebraic equations. The second step consists in calculating solutions for all right-hand sides. For reducing the communication interactions, based on the formulated and proved the main Gaussian Parallel Elimination Theorem for tridiagonal system of equations, we propose an original algorithm for calculating share components of the solution vector. Theoretical estimates validating the efficiency of the approach for both the common- and distributed-memory supercomputers are obtained. Direct and iterative methods of solving a 2D Poisson equation, which include procedures of tridiagonal matrix inversion, are realized using the MPI paradigm. Results of computational experiments on a multicomputer demonstrate a high efficiency and scalability of the parallel Dichotomy Algorithm.

A new algorithm for analytic element modeling of large-scale groundwater flow

This work introduces a new iterative Analytic Element Method (AEM) algorithm for solving 2D steady-state groundwater flow models containing large numbers of head-specified elements (e.g., rivers and lakes). The new algorithm improves convergence of models containing head-specified elements by explicitly computing fluxes of all such features at the start of each iteration. The new algorithm also enables the use of efficient parallel processing on distributed-memory super-computers. The combination of parallel processing and reduced number of iterations significantly extends the size and complexity of problems that can be modeled using AEM.