Piz Daint Research Articles

The ICON-A model for direct QBO simulations on GPUs (version icon-cscs:baf28a514)

Abstract. Classical numerical models for the global atmosphere, as used for numerical weather forecasting or climate research, have been developed for conventional central processing unit (CPU) architectures. This hinders the employment of such models on current top-performing supercomputers, which achieve their computing power with hybrid architectures, mostly using graphics processing units (GPUs). Thus also scientific applications of such models are restricted to the lesser computer power of CPUs. Here we present the development of a GPU-enabled version of the ICON atmosphere model (ICON-A), motivated by a research project on the quasi-biennial oscillation (QBO), a global-scale wind oscillation in the equatorial stratosphere that depends on a broad spectrum of atmospheric waves, which originates from tropical deep convection. Resolving the relevant scales, from a few kilometers to the size of the globe, is a formidable computational problem, which can only be realized now on top-performing supercomputers. This motivated porting ICON-A, in the specific configuration needed for the research project, in a first step to the GPU architecture of the Piz Daint computer at the Swiss National Supercomputing Centre and in a second step to the JUWELS Booster computer at the Forschungszentrum Jülich. On Piz Daint, the ported code achieves a single-node GPU vs. CPU speedup factor of 6.4 and allows for global experiments at a horizontal resolution of 5 km on 1024 computing nodes with 1 GPU per node with a turnover of 48 simulated days per day. On JUWELS Booster, the more modern hardware in combination with an upgraded code base allows for simulations at the same resolution on 128 computing nodes with 4 GPUs per node and a turnover of 133 simulated days per day. Additionally, the code still remains functional on CPUs, as is demonstrated by additional experiments on the Levante compute system at the German Climate Computing Center. While the application shows good weak scaling over the tested 16-fold increase in grid size and node count, making also higher resolved global simulations possible, the strong scaling on GPUs is relatively poor, which limits the options to increase turnover with more nodes. Initial experiments demonstrate that the ICON-A model can simulate downward-propagating QBO jets, which are driven by wave–mean flow interaction.

A scalable algorithm for many-body dissipative particle dynamics using multiple general purpose graphic processing units

We present a novel algorithm for the many-body Dissipative Particle Dynamics (DPD) forces calculation which allows to efficiently scale the DL_MESO software package on Multiple General Purpose Graphic Processing Units. Together with the extension to 64-bit integer arrays and addition of hard surface boundary conditions, the proposed algorithm allows to simulate very large complex mesoscale systems up to 14 billion beads. The implementation takes advantages of the CUDA language stream features to overlap the exchange of particle positions and local densities and the computation of the short range forces. We tested a water drop between two plates system using tree of the main European supercomputers: Piz Daint, Marconi and JUWELS. Results shows an improvement on the speedup compared to a naive implementation up to 1.5x when using 1024 GPUs.

Korali: Efficient and scalable software framework for Bayesian uncertainty quantification and stochastic optimization

We present Korali, an open-source framework for large-scale Bayesian uncertainty quantification and stochastic optimization. The framework relies on non-intrusive sampling of complex multiphysics models and enables their exploitation for optimization and decision-making. In addition, its distributed sampling engine makes efficient use of massively-parallel architectures while introducing novel fault tolerance and load balancing mechanisms. We demonstrate these features by interfacing Korali with existing high-performance software such as Aphros, LAMMPS (CPU-based), and Mirheo (GPU-based) and show efficient scaling for up to 512 nodes of the CSCS Piz Daint supercomputer. Finally, we present benchmarks demonstrating that Korali outperforms related state-of-the-art software frameworks.

The LISE package: Solvers for static and time-dependent superfluid local density approximation equations in three dimensions

Nuclear implementation of the density functional theory (DFT) is at present the only microscopic framework applicable to the whole nuclear landscape. The extension of DFT to superfluid systems in the spirit of the Kohn-Sham approach, the superfluid local density approximation (SLDA) and its extension to time-dependent situations, time-dependent superfluid local density approximation (TDSLDA), have been extensively used to describe various static and dynamical problems in nuclear physics, neutron star crust, and cold atom systems. In this paper, we present the codes that solve the static and time-dependent SLDA equations in three-dimensional coordinate space without any symmetry restriction. These codes are fully parallelized with the message passing interface (MPI) library and the time-dependent code takes advantage of graphic processing units (GPU) for accelerating execution. The dynamic code has checkpoint/restart capabilities and for initial conditions one can use any generalized Slater determinant type of wave function. By generating the appropriate initial quasi-particle wave-functions in a static calculation only, the time-dependent code can describe a large number of physical problems: nuclear fission, collisions of heavy ions, the interaction of quantized vortices with nuclei in the nuclear star crust, excitation of superfluid fermion systems by time dependent external fields, quantum shock waves, domain wall generation and propagation, the dynamics of the Anderson-Bogoliubov-Higgs mode, dynamics of fragmented condensates, vortex rings dynamics, generation and dynamics of quantized vortices, their crossing and recombinations and the incipient phases of quantum turbulence. Program summaryProgram title: LISECPC library link to program files:https://doi.org/10.17632/mwr7rsxpjw.1Licensing provisions: BSD 3-ClauseProgramming language: C, CUDANature of problem: The description of nuclear fission and nuclear reactions within the mean field approximation in real time within the extension of the density functional theory to superfluid systems is an extremely computationally demanding problem, which requires the solutions of a very large system of nonlinear coupled complex partial differential equations in 3+1 coordinates. Similar problems also appear in the case of cold atoms and in the dynamics of the neutron star crust, which have been tackled within the same framework with the same codes, using appropriate initial quasi-particle wave functions and/or an appropriate time-dependent external potential.Solution method: The evolution equations are discretized on a 3-dimensional spatial lattice and propagated in time. Spatial derivatives are evaluated using the fast Fourier transform technique. The propagation in time is performed using a predictor-modifier-corrector algorithm due to Adams-Bashforth-Milne, which requires only two evaluations of the right hand side of the equations per time step. The numerical accuracy of the time integration is ∼O(Δt)6. This method has a low truncation error, excellent numerical stability, and low roundoff errors.Additional comments including restrictions and unusual features: The code has been implemented on a variety of supercomputers (Jaguar, Titan, Piz Daint, Tsubame, Summit, Sierra) and demonstrates excellent scaling properties. The strong scaling capabilities can be significantly affected if a very large number of GPUs is used, when the communication time between processes overtakes the computation time as the dominant run time cost.

TinyMD: Mapping molecular dynamics simulations to heterogeneous hardware using partial evaluation

This paper investigates the suitability of the AnyDSL partial evaluation framework to implement tinyMD: an efficient, scalable, and portable simulation of pairwise interactions among particles. We compare tinyMD with the miniMD proxy application that scales very well on parallel supercomputers. We discuss the differences between both implementations and contrast miniMD’s performance for single-node CPU and GPU targets, as well as its scalability on SuperMUC-NG and Piz Daint supercomputers. Additionally, we demonstrate tinyMD’s flexibility by coupling it with the waLBerla multi-physics framework. This allow us to execute tinyMD simulations using the load-balancing mechanism implemented in waLBerla.

Highly efficient lattice Boltzmann multiphase simulations of immiscible fluids at high-density ratios on CPUs and GPUs through code generation

A high-performance implementation of a multiphase lattice Boltzmann method based on the conservative Allen-Cahn model supporting high-density ratios and high Reynolds numbers is presented. Meta-programming techniques are used to generate optimized code for CPUs and GPUs automatically. The coupled model is specified in a high-level symbolic description and optimized through automatic transformations. The memory footprint of the resulting algorithm is reduced through the fusion of compute kernels. A roofline analysis demonstrates the excellent efficiency of the generated code on a single GPU. The resulting single GPU code has been integrated into the multiphysics framework waLBerla to run massively parallel simulations on large domains. Communication hiding and GPUDirect-enabled MPI yield near-perfect scaling behavior. Scaling experiments are conducted on the Piz Daint supercomputer with up to 2048 GPUs, simulating several hundred fully resolved bubbles. Further, validation of the implementation is shown in a physically relevant scenario—a three-dimensional rising air bubble in water.

Dynamic Distribution of High-Rate Data Processing from CERN to Remote HPC Data Centers

The prompt reconstruction of the data recorded from the Large Hadron Collider (LHC) detectors has always been addressed by dedicated resources at the CERN Tier-0. Such workloads come in spikes due to the nature of the operation of the accelerator and in special high load occasions experiments have commissioned methods to distribute (spill-over) a fraction of the load to sites outside CERN. The present work demonstrates a new way of supporting the Tier-0 environment by provisioning resources elastically for such spilled-over workflows onto the Piz Daint Supercomputer at CSCS. This is implemented using containers, tuning the existing batch scheduler and reinforcing the scratch file system, while still using standard Grid middleware. ATLAS, CMS and CSCS have jointly run selected prompt data reconstruction on up to several thousand cores on Piz Daint into a shared environment, thereby probing the viability of the CSCS high performance computer site as on demand extension of the CERN Tier-0, which could play a role in addressing the future LHC computing challenges for the high luminosity LHC.

S-Acylation Controls SARS-CoV-2 Membrane Lipid Organization and Enhances Infectivity

SARS-CoV-2 virions are surrounded by a lipid bilayer which contains membrane proteins such as Spike, responsible for target-cell binding and virus fusion, the envelope protein E and the accessory protein Orf3a. Here, we show that during SARS-CoV-2 infection, all three proteins become lipid modified, through action of the S- acyltransferase ZDHHC20. Particularly striking is the rapid acylation of Spike on 10 cytosolic cysteines within the ER and Golgi. Using a combination of computational, lipidomics and biochemical approaches, we show that this massive lipidation controls Spike biogenesis and degradation, and drives the formation of localized ordered cholesterol and sphingolipid rich lipid nanodomains, in the early Golgi where viral budding occurs. ZDHHC20-mediated acylation allows the formation of viruses with enhanced fusion capacity and overall infectivity. Our study points towards S-acylating enzymes and lipid biosynthesis enzymes as novel therapeutic anti-viral targets.Funding: This work was supported by the Swiss National Science Foundation Corona Call, the CARIGEST foundation and the EPFL Corona Research task force to F.G.v.d.G. MD simulations were carried out on the Piz Daint computer at the Swiss Supercomputing Center (CSCS) thanks to access granted by PRACE Covid19 fast track project #17 (pr97) to M.D.P.Conflict of Interest: The authors declare no competing interests.

Digital blood in massively parallel CPU/GPU systems for the study of platelet transport.

We propose a highly versatile computational framework for the simulation of cellular blood flow focusing on extreme performance without compromising accuracy or complexity. The tool couples the lattice Boltzmann solver Palabos for the simulation of blood plasma, a novel finite-element method (FEM) solver for the resolution of deformable blood cells, and an immersed boundary method for the coupling of the two phases. The design of the tool supports hybrid CPU–GPU executions (fluid, fluid–solid interaction on CPUs, deformable bodies on GPUs), and is non-intrusive, as each of the three components can be replaced in a modular way. The FEM-based kernel for solid dynamics outperforms other FEM solvers and its performance is comparable to state-of-the-art mass–spring systems. We perform an exhaustive performance analysis on Piz Daint at the Swiss National Supercomputing Centre and provide case studies focused on platelet transport, implicitly validating the accuracy of our tool. The tests show that this versatile framework combines unprecedented accuracy with massive performance, rendering it suitable for upcoming exascale architectures.

Abelian–Higgs cosmic string evolution with multiple GPUs

Topological defects form at cosmological phase transitions by the Kibble mechanism. Cosmic strings and superstrings can lead to particularly interesting astrophysical and cosmological consequences, but this study is currently limited by the availability of accurate numerical simulations, which in turn is bottlenecked by hardware resources and computation time. Aiming to eliminate this bottleneck, in recent work we introduced and validated a GPU-accelerated evolution code for local Abelian–Higgs strings networks. While this leads to significant gains in speed, it is still limited by the physical memory available on a graphical accelerator. Here we report on a further step towards our main goal, by implementing and validating a multiple GPU extension of the earlier code, and further demonstrate its good scalability, both in terms of strong and weak scaling. A 81923 production run, using 4096 GPUs, runs in 33.2 min of wall-clock time on the Piz Daint supercomputer.

Ligand binding free-energy calculations with funnel metadynamics.

The accurate resolution of the binding mechanism of a ligand to its molecular target is fundamental to develop a successful drug design campaign. Free-energy calculations, which provide the energy value of the ligand-protein binding complex, are essential for resolving the binding mode of the ligand. The accuracy of free-energy calculation methods is counteracted by their poor user-friendliness, which hampers their broad application. Here we present the Funnel-Metadynamics Advanced Protocol (FMAP), which is a flexible and user-friendly graphical user interface (GUI)-based protocol to perform funnel metadynamics, a binding free-energy method that employs a funnel-shape restraint potential to reveal the ligand binding mode and accurately calculate the absolute ligand-protein binding free energy. FMAP guides the user through all phases of the free-energy calculation process, from preparation of the input files, to production simulation, to analysis of the results. FMAP delivers the ligand binding mode and the absolute protein-ligand binding free energy as outputs. Alternative binding modes and the role of waters are also elucidated, providing a detailed description of the ligand binding mechanism. The entire protocol on the paradigmatic system benzamidine-trypsin, composed of ~105 k atoms, took ~2.8 d using the Cray XC50 piz Daint cluster at the Swiss National Supercomputing Centre.

High-Dimensional Dynamic Stochastic Model Representation

We propose a generic and scalable method for computing global solutions of nonlinear, high-dimensional dynamic stochastic economic models. First, within an MPI–TBB parallel time-iteration framework, we approximate economic policy functions using an adaptive, high-dimensional model representation scheme, combined with adaptive sparse grids. With increasing dimensions, the number of points in this efficiently-chosen combination of low-dimensional grids grows much more slowly than standard tensor product grids, sparse grids, or even adaptive sparse grids. Moreover, the adaptivity within the individual component functions adds an additional layer of sparsity, since grid points are added only where they are most needed — that is to say, in regions of the computational domain with steep gradients or at non-differentiabilities. Second, we introduce a performant vectorization scheme of the interpolation compute kernel. Third, we validate our claims with numerical experiments conducted on “Piz Daint (Cray XC50) at the Swiss National Super-computing Center. We observe significant speedups over the state-of-the-art techniques, and almost ideal strong scaling up to at least 1, 000 compute nodes. Fourth, to demonstrate the broad applicability of our method, we compute global solutions to two different versions of a dynamic stochastic economic model: a high-dimensional international real business cycle model with capital adjustment costs, and with or without irreversible investment. We solve these models up to 300 continuous state variables globally.

A GPU-based algorithm for efficient LES of high Reynolds number flows in heterogeneous CPU/GPU supercomputers

Αn optimized MPI+OpenACC implementation model that performs efficiently in CPU/GPU systems using large-eddy simulation is presented. The code was validated for the simulation of wave boundary-layer flows against numerical and experimental data in the literature. A direct Fast-Fourier-Transform-based solver was developed for the solution of the Poisson equation for pressure taking advantage of the periodic boundary conditions. This solver was optimized for parallel execution in CPUs and outperforms by 10 times in computational time a typical iterative preconditioned conjugate gradient solver in GPUs. In terms of parallel performance, an overlapping strategy was developed to reduce the overhead of performing MPI communications using GPUs. As a result, the weak scaling of the algorithm was improved up to 30%. Finally, a large-scale simulation (Re = 2 × 105) using a grid of 4 × 108 cells was executed, and the performance of the code was analyzed. The simulation was launched using up to 512 nodes (512 GPUs + 6144 CPU-cores) on one of the current top 10 supercomputers of the world (Piz Daint). A comparison of the overall computational time showed that the GPU version was 4.2 times faster than the CPU one. The parallel efficiency of this strategy (47%) is competitive compared with the state-of-the-art CPU implementations, and it has the potential to take advantage of modern supercomputing capabilities.

Mirheo: High-performance mesoscale simulations for microfluidics

The transport and manipulation of particles and cells in microfluidic devices has become a core methodology in domains ranging from molecular biology to manufacturing and drug design. The design and operation of such devices can benefit from simulations that resolve flow-structure interactions at sub-micron resolution. We present a computational tool for large scale, efficient and high throughput mesoscale simulations of fluids and deformable objects at complex microscale geometries. The code employs dissipative particle dynamics for the description of the flow coupled with visco-elastic membrane model for red blood cells and can also handle rigid bodies and complex geometries. The software (Mirheo) is deployed on hybrid GPU/CPU architectures exhibiting unprecedented time-to-solution performance and excellent weak and strong scaling for a number of benchmark problems. Mirheo exploits the capabilities of GPU clusters, leading to speedup of up to 10X in terms of time to solution as compared to state-of-the-art software packages and reaches 90%–99% weak scaling efficiency on 512 nodes of the Piz Daint supercomputer. The software Mirheo relies on a Python interface to facilitate the solution and analysis of complex problems. Mirheo is an open source, potent computational tool that can greatly assist studies of microfluidics. Program summaryProgram Title: MirheoProgram Files doi:http://dx.doi.org/10.17632/n2dvz7htvn.1Licensing provisions: MITProgramming language: C++, CUDA, PythonNature of problem: 3D simulations of microfluidic flows in complex geometries with suspended rigid bodies and deformable membranes such as cells, bacteria and microparticles.Solution method: Dissipative particle dynamics are used to represent the fluid. Cell membrane dynamics are described through potentials for shear and bending energies that are discretized on a triangular mesh and by additional constraints on cell volume and membrane area. The model incorporates membrane viscosity and interactions between membranes and the surrounding fluid. Rigid objects and boundaries are represented by groups of particles with prescribed center of mass and rotation quaternion. Time integration is performed using the Velocity-Verlet algorithm.Additional comments including restrictions and unusual features: The code runs on Nvidia GPU accelerators starting with the Kepler generation.

Gyrokinetic simulations on many- and multi-core architectures with the global electromagnetic Particle-In-Cell Code ORB5

Gyrokinetic codes in plasma physics need outstanding computational resources to solve increasingly complex problems, requiring the effective exploitation of cutting-edge HPC architectures. This paper focuses on the enabling of ORB5, a state-of-the-art, first-principles-based gyrokinetic code, on modern parallel hybrid multi-core, multi-GPU systems. ORB5 is a Lagrangian, Particle-In-Cell (PIC), finite element, global, electromagnetic code, originally implementing distributed MPI-based parallelism through domain decomposition and domain cloning.In order to support multi/many cores devices, the code has been completely refactored. Data structures have been re-designed to ensure efficient memory access, enhancing data locality. Multi-threading has been introduced through OpenMP on the CPU and adopting OpenACC to support GPU acceleration. MPI is further used in combination with the two approaches. The performance results obtained using the full production ORB5 code on the Summit system at ORNL, on Piz Daint at CSCS and on the Marconi system at CINECA are presented, showing the effectiveness and performance portability of the adopted solutions: the same source code version was used to produce all results on all architectures.

Towards extreme scale dissipative particle dynamics simulations using multiple GPGPUs

A multi-GPGPU development for Mesoscale Simulations using the Dissipative Particle Dynamics method is presented. This distributed GPU acceleration development is an extension of the DL_MESO package to MPI+CUDA in order to exploit the computational power of the latest NVIDIA cards on hybrid CPU–GPU architectures. Details about the extensively applicable algorithm implementation and memory coalescing data structures are presented. The key algorithms’ optimizations for the nearest-neighbour list searching of particle pairs for short range forces, exchange of data and overlapping between computation and communications are also given. We have carried out strong and weak scaling performance analyses with up to 4096 GPUs. A two phase mixture separation test case with 1.8 billion particles has been run on the Piz Daint supercomputer from the Swiss National Supercomputer Center. With CUDA aware MPI, proper GPU affinity, communication and computation overlap optimizations for multi-GPU version, the final optimization results demonstrated more than 94% efficiency for weak scaling and more than 80% efficiency for strong scaling. As far as we know, this is the first report in the literature of DPD simulations being run on this large number of GPUs. The remaining challenges and future work are also discussed at the end of the paper.

Enabling ATLAS big data processing on Piz Daint at CSCS

Predictions for requirements for the LHC computing for Run 3 and Run 4 (HLLHC) over the course of the next 10 years show a considerable gap between required and available resources, assuming budgets will globally remain flat at best. This will require some radical changes to the computing models for the data processing of the LHC experiments. Concentrating computational resources in fewer larger and more efficient centres should increase the cost-efficiency of the operation and, thus, of the data processing. Large scale general purpose HPC centres could play a crucial role in such a model. We report on the technical challenges and solutions adopted to enable the processing of the ATLAS experiment data on the European flagship HPC Piz Daint at CSCS, now acting as a pledged WLCG Tier-2 centre. As the transition of the Tier-2 from classic to HPC resources has been finalised, we also report on performance figures over two years of production running and on efforts for a deeper integration of the HPC resource within the ATLAS computing framework at different tiers.

Lenstool-HPC: A High Performance Computing based mass modelling tool for cluster-scale gravitational lenses

With the upcoming generation of telescopes, cluster scale strong gravitational lenses will act as an increasingly relevant probe of cosmology and dark matter. The better resolved data produced by current and future facilities requires faster and more efficient lens modeling software. Consequently, we present Lenstool-HPC, a strong gravitational lens modeling and map generation tool based on High Performance Computing (HPC) techniques and the renowned Lenstool software. We also showcase the HPC concepts needed for astronomers to increase computation speed through massively parallel execution on supercomputers. Lenstool-HPC was developed using lens modelling algorithms with high amounts of parallelism. Each algorithm was implemented as a highly optimised CPU, GPU and Hybrid CPU-GPU version. The software was deployed and tested on the Piz Daint cluster of the Swiss National Supercomputing Centre (CSCS). Lenstool-HPC perfectly parallel lens map generation and derivative computation achieves a factor 30 speed-up using only 1 GPUs compared to Lenstool. Lenstool-HPC hybrid Lens-model fit generation tested at Hubble Space Telescope precision is scalable up to 200 CPU-GPU nodes and is faster than Lenstool using only 4 CPU-GPU nodes.

Heterogeneous Distributed Big Data Clustering on Sparse Grids

Clustering is an important task in data mining that has become more challenging due to the ever-increasing size of available datasets. To cope with these big data scenarios, a high-performance clustering approach is required. Sparse grid clustering is a density-based clustering method that uses a sparse grid density estimation as its central building block. The underlying density estimation approach enables the detection of clusters with non-convex shapes and without a predetermined number of clusters. In this work, we introduce a new distributed and performance-portable variant of the sparse grid clustering algorithm that is suited for big data settings. Our computed kernels were implemented in OpenCL to enable portability across a wide range of architectures. For distributed environments, we added a manager–worker scheme that was implemented using MPI. In experiments on two supercomputers, Piz Daint and Hazel Hen, with up to 100 million data points in a ten-dimensional dataset, we show the performance and scalability of our approach. The dataset with 100 million data points was clustered in 1198 s using 128 nodes of Piz Daint. This translates to an overall performance of 352 TFLOPS . On the node-level, we provide results for two GPUs, Nvidia’s Tesla P100 and the AMD FirePro W8100, and one processor-based platform that uses Intel Xeon E5-2680v3 processors. In these experiments, we achieved between 43% and 66% of the peak performance across all computed kernels and devices, demonstrating the performance portability of our approach.

DCA++: A software framework to solve correlated electron problems with modern quantum cluster methods

We present the first open release of the DCA++ project, a high-performance research software framework to solve quantum many-body problems with cutting edge quantum cluster algorithms. DCA++ implements the dynamical cluster approximation (DCA) and its DCA+ extension with a continuous self-energy. The algorithms capture nonlocal correlations in strongly correlated electron systems, thereby giving insight into high-Tc superconductivity. The code’s scalability allows efficient usage of systems at all scales, from workstations to leadership computers. With regard to the increasing heterogeneity of modern computing machines, DCA++ provides portable performance on conventional and emerging new architectures, such as hybrid CPU–GPU, sustaining multiple petaflops on ORNL’s Titan and CSCS’ Piz Daint supercomputers. Moreover, we show how sustainable and scalable development of the code base has been achieved by adopting standard techniques of the software industry. These include employing a distributed version control system, applying test-driven development and following continuous integration. Program summaryProgram Title: DCA++Program Files doi:http://dx.doi.org/10.17632/482jm5cv77.1Licensing provisions: BSD-3-ClauseProgramming language: C++14 and CUDANature of problem: Understanding the fascinating physics of strongly correlated electron systems requires the development of sophisticated algorithms and their implementation on leadership computing systems.Solution method: The DCA++ code provides a highly scalable and efficient implementation of the dynamical cluster approximation (DCA) and its DCA+ extension.