Hybrid Supercomputers Research Articles

Scale-resolving simulation of a low-pressure turbine on hybrid supercomputers

The increased computing power of modern hybrid supercomputers is expanding the applicability of scale-resolving simulations in scientific and industrial applications. This paper is devoted to such a supercomputer simulation technology for turbomachinery. A heterogeneous parallel algorithm is outlined and its parallel performance is demonstrated on various hybrid supercomputers by executing a single simulation on dozens of central and graphics processing units. The scale-resolving numerical simulation of the flow in a linear cascade of T106C high-lift low-pressure turbine blades is considered. The presence of a laminar–turbulent transition on the suction side of the blade, caused by the separation of the boundary layer, leads to a significant increase in the loss of kinetic energy. The effect of increasing losses with a decrease in the Reynolds number is captured. A strong dependence of the numerical results on the incoming turbulent flow conditions is observed. The effect of imposing synthetic turbulence at the inflow is studied, as well as the required mesh resolution to capture the laminar–turbulent transition well.

Adapting a Scientific CFD Code to Industrial Applications on Hybrid Supercomputers

The NOISEtte heterogeneous parallel code for simulating turbulent flow and aerodynamic noise is considered. In our previous works, high acceleration and parallel efficiency in scientific scale-resolving simulations using GPUs were reported. For parallelization, the MPI, OpenMP and OpenCL standards are used, the latter allows using GPUs from different vendors. However, the further transition to industrial-oriented applications brought more trouble. Instead of discussing the parallel algorithm, this work will focus on the problems that are not so obvious at first glance, which arise when developing a heterogeneous simulation code. How to deal with numerous simulation algorithm components, all those bells and whistles like wall functions, mixing plane and sliding interfaces, synthetic turbulence generators, a variety of boundary conditions, etc., that either need to be ported to the GPU side or incorporated directly from the CPU side? How to maintain and modify the OpenCL code in a growing number of source files? How to arrange the modularity of a complex heterogeneous software package? How to preserve reliability and fault tolerance, especially in the case of numerical schemes of increased accuracy, but reduced social responsibility? These issues are discussed here and some solutions will be proposed.

On the Parallelization of the Particle Method for Hybrid Supercomputers

On the Parallelization of the Particle Method for Hybrid Supercomputers

River Routing in the INM RAS-MSU Land Surface Model: Numerical Scheme and Parallel Implementation on Hybrid Supercomputers

The land surface model (LSM) is a necessary compartment of any numerical weather forecast system or the Earth system model. This paper presents a new version of the INM RAS-MSU land surface model where the river hydrodynamic and thermodynamic scheme is embedded into the parallel execution framework using MPI and OpenMP. Numerical experiments have been performed for the East European domain with resolution 0.5°× 0.5°. The soil model parallel efficiency at 1–144 MPI cores was 0.52–0.79 and limited by the presence of ocean area, and by imbalance of computational load between soil columns. The acceleration of the river model at MPI level was defined by the size of the largest river basin in the domain. At the OpenMP level, the potential for acceleration of large river basin simulation is shown to be close to number of threads used, based on fractal properties of the river networks. This acceleration was hindered in our numerical experiments by the reduced river orders at the coarse land surface model resolution, so that the optimal speedup for the Volga river basin was 2.5–3 times attained at 4–6 threads. This performance is projected to improve with refinement of the LSM spatial resolution.

Heterogeneous CPU+GPU parallelization for high-accuracy scale-resolving simulations of compressible turbulent flows on hybrid supercomputers

A heterogeneous parallel algorithm for simulation of compressible turbulent flows and its portable software implementation are presented. The underlying numerical method is based on a family of higher accuracy edge-based reconstruction schemes on unstructured mixed-element meshes. The proposed parallel solution can engage a large number of computing devices of most of the existing computing architectures used in modern supercomputers, including manycore CPUs and GPUs. It is capable of co-execution on both CPUs and accelerators simultaneously. The multilevel parallel algorithm combines: MPI for distributing workload among hybrid cluster nodes and between devices inside nodes; OpenMP for manycore CPUs and other supporting devices, such as Intel Xeon Phi; OpenCL for massively-parallel accelerators, such as GPUs of various vendors, including NVIDIA, AMD, Intel. The main focus is on the adaptation of the numerical method and its computational algorithm to the stream processing parallel paradigm. The very limited device memory inherent in GPU computing is also taken into account. A detailed description of the parallel algorithm is presented, as well as the techniques used for its efficient parallel implementation. Special attention is paid to implicit time integration with its linear solver and calculation of convective fluxes and viscous terms. The use of mixed floating-point precision and overlapping communications and computations is also discussed. Parallel performance is demonstrated in practical applications on different kinds of supercomputers using up to 10 thousand cores and multiple GPUs of comparable overall performance.

Simulations of plasmas and fluids using anti-symmetric models

ALMA (Anti-symmetric, Large-Moment, Accelerated) is a fast, flexible, and scalable toolkit designed to solve hyperbolic conservation law systems in hybrid supercomputers. This manuscript describes the theoretical background and implementation of ALMA, which uses the anti-symmetric formulation of fluids to obtain simple, robust, and easily paralellizable code. Practical GPU acceleration is realized on entire applications with an overall gain factor of 2 to 4. ALMA also provides a parallel, GPU accelerated sparse solver based on geometric multigrid, capable of diagonalizing linear systems with 239 unknowns. We demonstrate ALMA's scaling and performance in petascale supercomputers and use standard fluid models to verify the overall approach with canonical benchmark problems.

Load balancing method for heterogeneous CFD algorithms

Abstract. The problem of load balancing for unstructured heterogeneous numerical algorithms for simulation of physical processes is considered. A computational distribution method for hybrid supercomputers with multicore CPUs and massively parallel accelerators is described. The load balancing procedure includes determination of dual graph vertices and edges weights, devices’ performance test and two-level decomposition of the computational mesh based on domain decomposition method. First level decomposition involves the graph partitioning between supercomputer nodes. On the second level node subdomains are partitioned between the MPI- processes running on the nodes. The details of the proposed approach are considered on the example of an unstructured finite-volume algorithm for modeling the Navier-Stokes equations with polynomial reconstruction of variables and explicit time integration scheme. The parallel version of the algorithm is developed using the MPI, OpenMP and CUDA programming models. The parameters of performance, parallel efficiency and scalability of the heterogeneous program are given. The results mentioned are obtained during the simulation of a supersonic flow around a sphere on a mixed mesh consisting of tetrahedrons, triangular prisms, quadrangular pyramids and hexagons.

A hierarchical parallel implementation for heterogeneous computing. Application to algebra-based CFD simulations on hybrid supercomputers

The quest for new portable implementations of simulation algorithms is motivated by the increasing variety of computing architectures. Moreover, the hybridization of high-performance computing systems imposes additional constraints, since heterogeneous computations are needed to efficiently engage processors and massively-parallel accelerators. This, in turn, involves different parallel paradigms and computing frameworks and requires complex data exchanges between computing units. Typically, simulation codes rely on sophisticated data structures and computing subroutines, so-called kernels, which makes portability terribly cumbersome. Thus, a natural way to achieve portability is to dramatically reduce the complexity of both data structures and computing kernels. In our algebra-based approach, the scale-resolving simulation of incompressible turbulent flows on unstructured meshes relies on three fundamental kernels: the sparse matrix-vector product, the linear combination of vectors and the dot product. It is noteworthy that this approach is not limited to a particular kind of numerical method or a set of governing equations. In our code, an auto-balanced multilevel partitioning distributes workload among computing devices of various architectures. The overlap of computations and multistage communications efficiently hides the data exchanges overhead in large-scale supercomputer simulations. In addition to computing on accelerators, special attention is paid at efficiency on manycore processors in multiprocessor nodes with significant non-uniform memory access factor. Parallel efficiency and performance are studied in detail for different execution modes on various supercomputers using up to 9,600 processor cores and up to 256 graphics processor units. The heterogeneous implementation model described in this work is a general-purpose approach that is well suited for various subroutines in numerical simulation codes.

Cross-platform implementation of Particle-In-Cell method for simulation of high-temperature and fusion plasma by means of hybrid supercomputers equipped with GPU or Intel Xeon Phi accelerators

A new Python-based Particle-In-Cell code is presented. The code uses leapfrog particle pusher. The important feature of our code is that all the particles are pushed at once, thus the code is vectorized to improve performance. Electric field is given by Poisson equation with Least squares solver. The code involves collision simulation by PIC-MC method. Both large-scale (MPI) and fine-grain parallelization are being used. The implementation is based on the efficient NumPy library in Python language with the help of Dask package to improve Numpy performance. GPU implementation involves PyCUDA and the performance with Intel Xeon processors and Intel Xeon Phi accelerators is supported by high-performance Intel Python.

Parallel Algorithm for Supercomputing Simulation of Dust-Gaseous Gravitating Systems Using Particle-In-Cell and SPH Methods

We present a supercomputer numerical model for simulating 3D dynamics of dust-gas gravitating circumstellar disk. The model combines SPH method for solving gas dynamical equations, particle-in-cell (PIC) method for solving Vlasov equation, and convolution FFT-based grid method for solving Poisson equation. The approach is based on 3D domain decomposition, sorting of particles with regard to grid cell, and transferring necessary amount of particles between subdomains. The algorithm is aimed to be run on supercomputers with distributed memory and on hybrid supercomputers.

Heterogeneous Computing in Resource-Intensive CFD Simulations

A software package implementing a fully heterogeneous mode of computations on CPUs and GPU accelerators for efficient use of hybrid supercomputers has been developed at the Keldysh Institute of Applied Mathematics of the Russian Academy of Sciences. The package involves a distributed preprocessor ensuring work with fine unstructured meshes. Combined compression of the grid topology is used to reduce the amount of storage required for superlarge grid data. The study involves petascale computational resources.

Multilevel parallelization for simulating compressible turbulent flows on most kinds of hybrid supercomputers

The paper describes a multilevel MPI+OpenMP+OpenCL parallelization approach that provides complete portability across a wide range of hybrid supercomputer architectures. A parallel CFD algorithm for heterogeneous computing of turbulent flows is presented. It simulates the compressible Navier–Stokes equations using a cell-centered finite-volume method with polynomial reconstruction on unstructured hybrid meshes. A two-level partitioning is used for the workload distribution among computing devices of hybrid nodes. The overlap of communications and computations hides the data transfer expenses. The scalability is tested on various HPC systems including a fat node with 8 GPUs and supercomputers using up to 320 GPUs. Comparison of performance is presented for multicore CPUs, Intel Xeon Phi, various GPUs of AMD and NVIDIA. The heterogeneous execution using CPUs and GPUs is studied in detail.

A New Hydrodynamic Model for Numerical Simulation of Interacting Galaxies on Intel Xeon Phi Supercomputers

This paper presents a new hydrodynamic model of interacting galaxies based on the joint solution of multicomponent hydrodynamic equations, first moments of the collisionless Boltzmann equation and the Poisson equation for gravity. Using this model, it is possible to formulate a unified numerical method for solving hyperbolic equations. This numerical method has been implemented for hybrid supercomputers with Intel Xeon Phi accelerators. The collision of spiral and disk galaxies considering the star formation process, supernova feedback and molecular hydrogen formation is shown as a simulation result.

Адаптивный анализ надежности паролей при помощи гибридных суперЭВМ

Адаптивный анализ надежности паролей при помощи гибридных суперЭВМ

AstroPhi: A code for complex simulation of the dynamics of astrophysical objects using hybrid supercomputers

We propose a new code named AstroPhi for simulation of the dynamics of astrophysical objects on hybrid supercomputers equipped with Intel Xenon Phi computation accelerators. The details of parallel implementation are described, as well as changes to the computational algorithm that facilitate efficient parallel implementation. A single Xeon Phi accelerator yielded 27-fold acceleration. The use of 32 Xeon Phi accelerators resulted in 94% parallel efficiency. Several collapse problems are simulated using the AstroPhi code. Program summaryProgram title: AstroPhiCatalogue identifier: AEUM_v1_0Program summary URL:http://cpc.cs.qub.ac.uk/summaries/AEUM_v1_0.htmlProgram obtainable from: CPC Program Library, Queen’s University, Belfast, N. IrelandLicensing provisions: Standard CPC licence, http://cpc.cs.qub.ac.uk/licence/licence.htmlNo. of lines in distributed program, including test data, etc.: 99604No. of bytes in distributed program, including test data, etc.: 305433Distribution format: tar.gzProgramming language: C++.Computer: MVS-10P - RSC Tornado, Xeon E5-2690 8C 2.900 GHz, Infiniband FDR, Intel Xeon Phi SE10X.Operating system: Linux.Has the code been vectorized or parallelized?: Parallelized on MPI + OpenMP for Intel MIC architecture, 32 Intel Xeon Phi (60 cores per 1 Intel Xeon Phi = 1920 cores of Intel Xeon Phi).RAM: 137438953472 bytes (128 GB) bytesClassification: 1.9.External routines: MPI, OpenMP for Intel Xeon Phi, FFTW 2.1.5Nature of problem:Complex numerical simulation of dynamics of astrophysical objects plays an important role due to significant growth of observational astronomic data. The new astrophysical models and codes need to be developed for detailed simulation of different physical effects in astrophysics with the use of modern supercomputers with hybrid architecture.Solution method:AstroPhi code consisting of particle-in-cell and Godunov methods combination adapted for hybrid supercomputer architecture.Restrictions:For this version maximum grid size is restricted to 10243.Running time:Typical running on MVS-10P is 24 h. The test provided only takes a few minutes.

Co-design of Parallel Numerical Methods for Plasma Physics and Astrophysics

Physically meaningful simulations in plasma physics and astrophysics need powerful hybrid supercomputers equipped with computation accelerators. The development of parallel numerical codes for such supercomputers is a complex scientific problem. In order to solve it the concept of codesign is employed. The codesign is defined as considering the architecture of the supercomputer at all stages of the development of the code. The use of codesign is shown by the example of two physical problems: the interaction of an electron beam with plasma and the collision of galaxies. The efficiency is 92 % with 500 Tesla GPUs at the Lomonosov supercomputer. The test computation involved 160 million of model particles.

Эффективное использование многоядерных сопроцессоров при суперкомпьютерном статистическом моделировании электронных лавин

Three-dimensional parallel Monte Carlo algorithm for modelling the electron avalanches in gases is developed. Parallel Implementation is made on supercomputers with MPP architecture and on hybrid supercomputers with Intel Xeon Phi coprocessors. The well-working library PARMONC is used to implement parallel computations. The use of the library enables fast calculation of functionals such as the number of particles in avalanche, first Townsend coefficient, drift velocity, etc.

A parallel MPI + OpenMP + OpenCL algorithm for hybrid supercomputations of incompressible flows

The work is devoted to the development of efficient parallel algorithms for large-scale simulations of incompressible flows on hybrid supercomputers based on massively-parallel accelerators. The governing equations are discretized using a high-order finite-volume scheme for Cartesian staggered meshes with the only restriction that, at least, one direction is periodic. Its “classical” MPI+OpenMP parallel implementation for CPUs was designed to scale till 100,000 CPU cores. The new hybrid algorithm is developed on a base of a multi-level parallel model that exploits several layers of parallelism of a modern hybrid supercomputer. In this model, MPI and OpenMP are used on the first two levels to couple nodes of a supercomputer and to engage its CPU cores. Then, computing accelerators are further used by means of the hardware independent OpenCL computing standard. In this way, the implementation is adapted to a general computing model with central processors and math co-processors. In this paper the work is focused on adapting the basic operations of the algorithm to architectures of Graphics Processing Units (GPU) without considering the multi-GPU communication scheme. Technology of porting the code to OpenCL is described, certain optimization approaches are presented and relevant performance results obtaining up to 80–90 GFLOPS on a GPU accelerator are demonstrated.Moreover, the experience with different GPU architectures is summarized and a comparison based on the particular application is given for AMD and NVIDIA GPUs as well as for CUDA and OpenCL frameworks.

An OpenCL-based Parallel CFD Code for Simulations on Hybrid Systems with Massively-parallel Accelerators

A parallel finite-volume CFD algorithm for modeling of incompressible flows on hybrid supercomputers is presented. It is based on a symmetry-preserving high-order numerical scheme for structured meshes. A multilevel approach that combines different parallel models is used for large-scale simulations on computing systems with massively-parallel accelerators. MPI is used on the first level within the distributed memory model to couple computing nodes of a supercomputer. On the second level OpenMP is used to engage multiple CPU cores of a computing node. The third level exploits the computing potential of massively-parallel accelerators such as GPU (Graphics Processing Units) of AMD and NVIDIA, or Intel Xeon Phi accelerators of the MIC (Many Integrated Core) architecture. The hardware independent OpenCL standard is used to compute on accelerators of different archi- tectures within a general model for a combination of a central processor and a math co-processor.

Direct Numerical Simulation of Incompressible Flows on Unstructured Meshes Using Hybrid CPU/GPU Supercomputers

This paper describes a hybrid MPI-CUDA parallelization strategy for the direct numerical simulation of incompressible flows using unstructured meshes. Our in-house MPI-based unstructured CFD code has been extended in order to increase its performance by means of GPU co-processors. Therefore, the main goal of this work is to take advantage of the current hybrid supercomputers to increase our computing capabilities. CUDA is used to perform the calculations on the GPU devices and MPI to handle the communications between them. The main drawback for the performance is the slowdown produced by the MPI communication episodes. Consequently, overlapping strategies, to hide MPI communication costs under GPU computations, are studied in detail with the aim to achieve scalability when executing the code on multiple nodes.