Supercomputer Architectures Research Articles

Scalability investigation of parallel algorithms for plasma dynamics problems in open magnetic traps by simulation modeling

This paper describes the use of predictive simulation modeling for investigating the scalability of parallel algorithms. The study is carried out using a multiagent approach. The general principles of designing a model for the execution of a parallel algorithm on a given supercomputer architecture are described along with their application for researching this algorithm. The paper also presents the results of simulating the execution of this algorithm on more than a million cores.

Identification of stable adsorption sites and diffusion paths on nanocluster surfaces: an automated scanning algorithm

The diverse coordination environments on the surfaces of discrete, three-dimensional (3D) nanoclusters contribute significantly to their unique catalytic properties. Identifying the numerous adsorption sites and diffusion paths on these clusters is however tedious and time-consuming, especially for large, asymmetric nanoclusters. Here, we present a simple, automated method for constructing approximate 2D potential energy surfaces for the adsorption of atomic species on the surfaces of 3D nanoclusters with minimal human intervention. These potential energy surfaces fully characterize the important adsorption sites and diffusion paths on the nanocluster surfaces with accuracies similar to current approaches and at comparable computational cost. Our method can treat complex nanoclusters, such as alloy nanoclusters, and accounts for cluster relaxation and adsorbate-induced reconstruction, important for obtaining accurate energetics. Moreover, its highly parallelizable nature is ideal for modern supercomputer architectures. We showcase our method using two clusters: Au18 and Pt55. For Au18, diffusion of atomic hydrogen between the most stable sites occurs via non-intuitive paths, underlining the necessity of exploring the complete potential energy surface. By enabling the rapid and unbiased assessment of adsorption and diffusion on large, complex nanoclusters, which are particularly difficult to handle manually, our method will help advance materials discovery and the rational design of catalysts.

GAMERA: A Three-dimensional Finite-volume MHD Solver for Non-orthogonal Curvilinear Geometries

Abstract Efficient simulation of plasmas in various contexts often involves the use of meshes that conform to the intrinsic geometry of the system under consideration. We present here a description of a new magnetohydrodynamic code, GAMERA (Grid Agnostic MHD for Extended Research Applications), designed to combine geometric flexibility with high-order spatial reconstruction and constrained transport to maintain the divergence-free magnetic field. GAMERA carries on the legacy of its predecessor, the LFM (Lyon–Fedder–Mobarry), a research code whose use in space physics has spanned three decades. At the time of its initial development, the LFM code had a number of novel features: eighth-order centered spatial differencing, the Partial Donor Cell Method limiter for shock capturing, a non-orthogonal staggered mesh with constrained transport, and conservative averaging-reconstruction for axis singularities. The capability to handle multiple ion species was also added later. GAMERA preserves the core numerical philosophy of LFM while also incorporating numerous algorithmic and computational improvements. The upgrades in the numerical schemes include accurate grid metric calculations using high-order Gaussian quadrature techniques, high-order upwind reconstruction, non-clipping options for interface values, and improved treatment of axis singularities. The improvements in the code implementation include the use of data structures and memory access patterns conducive to aligned vector operations and the implementation of hybrid parallelism, using MPI and OMP. GAMERA is designed to be a portable and easy-to-use code that implements multidimensional MHD simulations in arbitrary non-orthogonal curvilinear geometries on modern supercomputer architectures.

An efficient real-time data collection framework on petascale systems

High efficiency data collection remains a great challenge for HPC reliability and resilience, yet may pave the way to overcome the barrier before fault prediction. Not only for increasing scalability up to exascale systems but even for contemporary supercomputer architectures does it require substantial efforts to efficiently collect and analyze data that contains system fault information within the framework of faults prediction. In this term, the article mainly focuses on efficient data collection and data preprocessing, preferring to optimize an effective framework to improve the efficiency of data collection in petascale system. The core of our framework includes a data collection acceleration layer scheduled by H2FS, a further detailed information get by performance analysis tool, as well as a new method for log template extraction, which all attribute to a more efficient and convenient framework for the real-time data collection. Hereafter, we conducted extensive tests based on a petascale system to verify the solution, and the experimental results demonstrate to be effectiveness and scalability of our framework.

Comparison of the capabilities of GPU clusters and general-purpose supercomputers for solving 3D inverse problems of ultrasound tomography

This study focuses on developing the algorithms for solving 3D inverse problems of ultrasound tomography using GPU clusters and general-purpose supercomputers. The computing capabilities of these supercomputer architectures in application to wave tomography are compared via numerical simulations.Parallel gradient-based iterative algorithms designed to solve coefficient inverse problems for the wave equation are proposed. We show that layer-by-layer wave tomography can be efficiently performed using both CPU- and GPU-based supercomputers. Solving 3D coefficient inverse problems is much more computationally expensive. A general-purpose (CPU-based) system capable of reconstructing 3D images should be equipped with either a sufficient number of memory access channels or a large number of computing cores with large cache memory and high-speed communication network. Such a system would be very large and expensive. Therefore, the most promising solution is to develop dedicated GPU-based supercomputers.From the applications perspective, this study focuses on problems of developing ultrasound tomography devices for breast cancer diagnosis. Computer simulations show that a GPU cluster capable of performing 3D image reconstruction within reasonable time fits in a single rack and can be incorporated into medical ultrasound tomography facilities.

Planetary giant impacts: convergence of high-resolution simulations using efficient spherical initial conditions and swift

ABSTRACT We perform simulations of giant impacts on to the young Uranus using smoothed particle hydrodynamics (SPH) with over 100 million particles. This 100–1000 × improvement in particle number reveals that simulations with below 107 particles fail to converge on even bulk properties such as the post-impact rotation period, or on the detailed erosion of the atmosphere. Higher resolutions appear to determine these large-scale results reliably, but even 108 particles may not be sufficient to study the detailed composition of the debris – finding that almost an order of magnitude more rock is ejected beyond the Roche radius than with 105 particles. We present two software developments that enable this increase in the feasible number of particles. First, we present an algorithm to place any number of particles in a spherical shell such that they all have an SPH density within 1 per cent of the desired value. Particles in model planets built from these nested shells have a root-mean-squared velocity below 1 per cent of the escape speed, which avoids the need for long precursor simulations to produce relaxed initial conditions. Secondly, we develop the hydrodynamics code sph with interdependent fine-grained tasking(swift) for planetary simulations. swift uses task-based parallelism and other modern algorithmic approaches to take full advantage of contemporary supercomputer architectures. Both the particle placement code and swift are publicly released.

Quantum Package 2.0: An Open-Source Determinant-Driven Suite of Programs.

Quantum chemistry is a discipline which relies heavily on very expensive numerical computations. The scaling of correlated wave function methods lies, in their standard implementation, between and , where N is proportional to the system size. Therefore, performing accurate calculations on chemically meaningful systems requires (i) approximations that can lower the computational scaling and (ii) efficient implementations that take advantage of modern massively parallel architectures. Quantum Package is an open-source programming environment for quantum chemistry specially designed for wave function methods. Its main goal is the development of determinant-driven selected configuration interaction (sCI) methods and multireference second-order perturbation theory (PT2). The determinant-driven framework allows the programmer to include any arbitrary set of determinants in the reference space, hence providing greater methodological freedom. The sCI method implemented in Quantum Package is based on the CIPSI (Configuration Interaction using a Perturbative Selection made Iteratively) algorithm which complements the variational sCI energy with a PT2 correction. Additional external plugins have been recently added to perform calculations with multireference coupled cluster theory and range-separated density-functional theory. All the programs are developed with the IRPF90 code generator, which simplifies collaborative work and the development of new features. Quantum Package strives to allow easy implementation and experimentation of new methods, while making parallel computation as simple and efficient as possible on modern supercomputer architectures. Currently, the code enables, routinely, to realize runs on roughly 2 000 CPU cores, with tens of millions of determinants in the reference space. Moreover, we have been able to push up to 12 288 cores in order to test its parallel efficiency. In the present manuscript, we also introduce some key new developments: (i) a renormalized second-order perturbative correction for efficient extrapolation to the full CI limit and (ii) a stochastic version of the CIPSI selection performed simultaneously to the PT2 calculation at no extra cost.

TopoLens: Building a CyberGIS community data service for enhancing the usability of high‐resolution national topographic datasets

SummaryIn recent years, geospatial data have exploded to massive volume and diversity and subsequently cause serious usability issues for researchers in various scientific areas. This paper describes a cyberGIS community data service framework to facilitate geospatial big data access, processing, and sharing based on a hybrid supercomputer architecture. Specifically, the framework aims to enhance the usability of national elevation dataset released by the U.S. Geological Survey in the contiguous United States at the resolution of arc‐second. A community data service, namely TopoLens, is created to demonstrate the workflow integration of national elevation dataset and the associated computation and analysis. Two user‐friendly environments, including a publicly available web application and a private workspace based on the Jupyter notebook, are provided for users to access both precomputed and on‐demand computed high‐resolution elevation data. The system architecture of TopoLens is implemented by exploiting the ROGER supercomputer, the first cyberGIS supercomputer dedicated to geospatial problem‐solving. The usability of TopoLens has been acknowledged in the topographic user community evaluation.

A CyberGIS Integration and Computation Framework for High‐Resolution Continental‐Scale Flood Inundation Mapping

AbstractWe present a Digital Elevation Model‐based hydrologic analysis methodology for continental flood inundation mapping (CFIM), implemented as a cyberGIS scientific workflow in which a 1/3rd arc‐second (10 m) height above nearest drainage (HAND) raster data for the conterminous United States (CONUS) was computed and employed for subsequent inundation mapping. A cyberGIS framework was developed to enable spatiotemporal integration and scalable computing of the entire inundation mapping process on a hybrid supercomputing architecture. The first 1/3rd arc‐second CONUS HAND raster dataset was computed in 1.5 days on the cyberGIS Resourcing Open Geospatial Education and Research supercomputer. The inundation mapping process developed in our exploratory study couples HAND with National Water Model forecast data to enable near real‐time inundation forecasts for CONUS. The computational performance of HAND and the inundation mapping process were profiled to gain insights into the computational characteristics in high‐performance parallel computing scenarios. The establishment of the CFIM computational framework has broad and significant research implications that may lead to further development and improvement of flood inundation mapping methodologies.

Multilevel parallelization for simulating compressible turbulent flows on most kinds of hybrid supercomputers

The paper describes a multilevel MPI+OpenMP+OpenCL parallelization approach that provides complete portability across a wide range of hybrid supercomputer architectures. A parallel CFD algorithm for heterogeneous computing of turbulent flows is presented. It simulates the compressible Navier–Stokes equations using a cell-centered finite-volume method with polynomial reconstruction on unstructured hybrid meshes. A two-level partitioning is used for the workload distribution among computing devices of hybrid nodes. The overlap of communications and computations hides the data transfer expenses. The scalability is tested on various HPC systems including a fat node with 8 GPUs and supercomputers using up to 320 GPUs. Comparison of performance is presented for multicore CPUs, Intel Xeon Phi, various GPUs of AMD and NVIDIA. The heterogeneous execution using CPUs and GPUs is studied in detail.

MILC Code Performance on High End CPU and GPU Supercomputer Clusters

With recent developments in parallel supercomputing architecture, many core, multi-core, and GPU processors are now commonplace, resulting in more levels of parallelism, memory hierarchy, and programming complexity. It has been necessary to adapt the MILC code to these new processors starting with NVIDIA GPUs, and more recently, the Intel Xeon Phi processors. We report on our efforts to port and optimize our code for the Intel Knights Landing architecture. We consider performance of the MILC code with MPI and OpenMP, and optimizations with QOPQDP and QPhiX. For the latter approach, we concentrate on the staggered conjugate gradient and gauge force. We also consider performance on recent NVIDIA GPUs using the QUDA library.

Constant-pH Molecular Dynamics Simulations for Large Biomolecular Systems.

An increasingly important endeavor is to develop computational strategies that enable molecular dynamics (MD) simulations of biomolecular systems with spontaneous changes in protonation states under conditions of constant pH. The present work describes our efforts to implement the powerful constant-pH MD simulation method, based on a hybrid nonequilibrium MD/Monte Carlo (neMD/MC) technique within the highly scalable program NAMD. The constant-pH hybrid neMD/MC method has several appealing features; it samples the correct semigrand canonical ensemble rigorously, the computational cost increases linearly with the number of titratable sites, and it is applicable to explicit solvent simulations. The present implementation of the constant-pH hybrid neMD/MC in NAMD is designed to handle a wide range of biomolecular systems with no constraints on the choice of force field. Furthermore, the sampling efficiency can be adaptively improved on-the-fly by adjusting algorithmic parameters during the simulation. Illustrative examples emphasizing medium- and large-scale applications on next-generation supercomputing architectures are provided.

ELSI: A unified software interface for Kohn–Sham electronic structure solvers

Solving the electronic structure from a generalized or standard eigenproblem is often the bottleneck in large scale calculations based on Kohn–Sham density-functional theory. This problem must be addressed by essentially all current electronic structure codes, based on similar matrix expressions, and by high-performance computation. We here present a unified software interface, ELSI, to access different strategies that address the Kohn–Sham eigenvalue problem. Currently supported algorithms include the dense generalized eigensolver library ELPA, the orbital minimization method implemented in libOMM, and the pole expansion and selected inversion (PEXSI) approach with lower computational complexity for semilocal density functionals. The ELSI interface aims to simplify the implementation and optimal use of the different strategies, by offering (a) a unified software framework designed for the electronic structure solvers in Kohn–Sham density-functional theory; (b) reasonable default parameters for a chosen solver; (c) automatic conversion between input and internal working matrix formats, and in the future (d) recommendation of the optimal solver depending on the specific problem. Comparative benchmarks are shown for system sizes up to 11,520 atoms (172,800 basis functions) on distributed memory supercomputing architectures. Program summaryProgram title: ELSI InterfaceProgram Files doi:http://dx.doi.org/10.17632/y8vzhzdm62.1Licensing provisions: BSD 3-clauseProgramming language: Fortran 2003, with interface to C/C++External routines/libraries: MPI, BLAS, LAPACK, ScaLAPACK, ELPA, libOMM, PEXSI, ParMETIS, SuperLU_DISTNature of problem: Solving the electronic structure from a generalized or standard eigenvalue problem in calculations based on Kohn–Sham density functional theory (KS-DFT).Solution method: To connect the KS-DFT codes and the KS electronic structure solvers, ELSI provides a unified software interface with reasonable default parameters, hierarchical control over the interface and the solvers, and automatic conversions between input and internal working matrix formats. Supported solvers are: ELPA (dense generalized eigensolver), libOMM (orbital minimization method), and PEXSI (pole expansion and selected inversion method).Restrictions: The ELSI interface requires complete information of the Hamiltonian matrix.

A Blocked Linear Method for Optimizing Large Parameter Sets in Variational Monte Carlo.

We present a modification to variational Monte Carlo's linear method optimization scheme that addresses a critical memory bottleneck while maintaining compatibility with both the traditional ground state variational principle and our recently introduced variational principle for excited states. For wave function ansatzes with tens of thousands of variables, our modification reduces the required memory per parallel process from tens of gigabytes to hundreds of megabytes, making the methodology a much better fit for modern supercomputer architectures in which data communication and per-process memory consumption are primary concerns. We verify the efficacy of the new optimization scheme in small molecule tests involving both the Hilbert space Jastrow antisymmetric geminal power ansatz and real space multi-Slater Jastrow expansions. Satisfied with its performance, we have added the optimizer to the QMCPACK software package, with which we test a systematically convergent, nonperturbative approach to excitation energies on the example of a Mott-insulating hydrogen ring.

LB3D: A parallel implementation of the Lattice-Boltzmann method for simulation of interacting amphiphilic fluids

We introduce the lattice-Boltzmann code LB3D, version 7.1. Building on a parallel program and supporting tools which have enabled research utilising high performance computing resources for nearly two decades, LB3D version 7 provides a subset of the research code functionality as an open source project. Here, we describe the theoretical basis of the algorithm as well as computational aspects of the implementation. The software package is validated against simulations of meso-phases resulting from self-assembly in ternary fluid mixtures comprising immiscible and amphiphilic components such as water–oil–surfactant systems. The impact of the surfactant species on the dynamics of spinodal decomposition are tested and quantitative measurement of the permeability of a body centred cubic (BCC) model porous medium for a simple binary mixture is described. Single-core performance and scaling behaviour of the code are reported for simulations on current supercomputer architectures. Program summaryProgram Title: LB3DProgram Files doi:http://dx.doi.org/10.17632/9g9x2wr8z8.1Licensing provisions: BSD 3-clauseProgramming language: FORTRAN90, Python, CNature of problem: Solution of the hydrodynamics of single phase, binary immiscible and ternary amphiphilic fluids. Simulation of fluid mixtures comprising miscible and immiscible fluid components as well as amphiphilic species on the mesoscopic scale. Observable phenomena include self-organisation of mesoscopic complex fluid phases and fluid transport in porous media.Solution method: Lattice-Boltzmann (lattice-Bhatnagar–Gross–Krook, LBGK) [1, 2, 3] method describing fluid dynamics in terms of the single particle velocity distribution function in a 3-dimensional discrete phase space (D3Q19) [4, 5, 6]. Multiphase interactions are modelled using a phenomenological pseudo-potential approach [7, 8] with amphiphilic interactions utilising an additional dipole field [9, 10]. Solid boundaries are modelled using simple bounce-back boundary conditions and additional pseudo-potential wetting interactions [11].Additional comments including Restrictions and Unusual features: The purpose of the release is the provision of a refactored minimal version of LB3D suitable as a starting point for the integration of additional features building on the parallel computation and IO functionality. [1]S. Succi, The Lattice Boltzmann Equation: For Fluid Dynamics and Beyond, Oxford University Press, 2001.[2]B. Dünweg, A. Ladd, Lattice Boltzmann simulations of soft matter systems, Adv. Poly. Sci. 221 (2009) 89–166[3]C. K. Aidun, J. R. Clausen, Lattice-Boltzmann Method for Complex Flows, Annual Review of Fluid Mechanics 42 (2010) 439.[4]X. He, L.-S. Luo, A priori derivation of the lattice-Boltzmann equation, Phys. Rev. E 55 (1997) R6333.[5]X. He, L.-S. Luo, Theory of the lattice Boltzmann method: from the Boltzmann equation to the lattice Boltzmann equation, Phys. Rev. E 56.[6]Y. H. Qian, D. D’Humiéres, P. Lallemand, Lattice BGK Models for Navier–Stokes Equation, Europhysics Letters 17 (1992) 479.[7]X. Shan, H. Chen, Lattice-Boltzmann model for simulating flows with multiple phases and components, Physical Review E 47 (1993) 1815.[8]X. Shan, G. Doolen, Multicomponent lattice-Boltzmann model with interparticle interaction, Journal of Statistical Physics 81 (1995) 379.[9]H. Chen, B. Boghosian, P.V. Coveney, M. Nekovee, A ternary lattice-Boltzmann model for amphiphilic fluids, Proceedings of the Royal Society of London A 456 (2000) 2043.[10]M. Nekovee, P. V. Coveney, H. Chen, B. M. Boghosian, Lattice-Boltzmann model for interacting amphiphilic fluids, Phys. Rev. E 62 (2000) 8282.[11]N. S. Martys, H. Chen, Simulation of multicomponent fluids in complex three-dimensional geometries by the lattice-Boltzmann method, Phys. Rev. E 53 (1996) 743.

Reproducible MPI Benchmarking is Still Not as Easy as You Think

The Message Passing Interface (MPI) is the prevalent programming model used on today's supercomputers. Therefore, MPI library developers are looking for the best possible performance (shortest run-time) of individual MPI functions across many different supercomputer architectures. Several MPI benchmark suites have been developed to assess the performance of MPI implementations. Unfortunately, the outcome of these benchmarks is often neither reproducible nor statistically sound. To overcome these issues, we show which experimental factors have an impact on the run-time of blocking collective MPI operations and how to measure their effect. Finally, we present a new experimental method that allows us to obtain reproducible and statistically sound measurements of MPI functions.

Methodology for the Simulation of Molecular Motors at Different Scales.

Millisecond-scale conformational transitions represent a seminal challenge for traditional molecular dynamics simulations, even with the help of high-end supercomputer architectures. Such events are particularly relevant to the study of molecular motors-proteins or abiological constructs that convert chemical energy into mechanical work. Here, we present a hybrid-simulation scheme combining an array of methods including elastic network models, transition path sampling, and advanced free-energy methods, possibly in conjunction with generalized-ensemble schemes to deliver a viable option for capturing the millisecond-scale motor steps of biological motors. The methodology is already implemented in large measure in popular molecular dynamics programs, and it can leverage the massively parallel capabilities of petascale computers. The applicability of the hybrid method is demonstrated with two examples, namely cyclodextrin-based motors and V-type ATPases.

Performance modeling of hyper-scale custom machine for the principal steps in block Wiedemann algorithm

Solving large-scale sparse linear systems over GF(2) plays a key role in fluid mechanics, simulation and design of materials, petroleum seismic data processing, numerical weather prediction, computational electromagnetics, and numerical simulation of unclear explosions. Therefore, developing algorithms for this issue is a significant research topic. In this paper, we proposed a hyper-scale custom supercomputer architecture that matches specific data features to process the key procedure of block Wiedemann algorithm and its parallel algorithm on the custom machine. To increase the computation, communication, and storage performance, four optimization strategies are proposed. This paper builds a performance model to evaluate the execution performance and power consumption for our custom machine. The model shows that the optimization strategies result in a considerable speedup, even three times faster than the fastest supercomputer, TH2, while consuming less power.

Spin-Unrestricted Second-Order Møller-Plesset (MP2) Forces for the Condensed Phase: From Molecular Radicals to F-Centers in Solids.

To obtain consistent geometries for the computation of properties, nuclear gradients are essential. Here, we report a fully periodic Γ-point, massively parallel implementation of spin-unrestricted second-order Møller-Plesset (MP2) forces. It is based on the resolution-of-identity and Gaussian and plane waves approach to calculate electron repulsion integrals and is made available in the CP2K program. The algorithm is optimized for modern supercomputer architectures and is capable of employing both CPUs and GPUs. The asymptotic computational scaling is O(N(5)), which is observed for the systems containing more than 100 second-row atoms with triple-ζ quality basis sets. For smaller systems, the energy and forces can be evaluated within minutes; for bigger systems within hours, provided that thousands of processor units are available. Spin-unrestricted MP2 calculations are, computationally, ∼3 times more demanding than spin-restricted ones, but exhibit a better parallel performance. As a bonus, the gradient implementation allows for computing spin-density distributions not available in energy calculations. The method has been successfully used for computing the crystal structure of the TEMPO radical and to obtain the spin density distribution of an F-center in lithium fluoride at a consistent geometry. We expect this computationally efficient tool to be useful for the chemistry, materials science, and solid-state physics communities.

Hybrid MPI-thread parallelization of adaptive mesh operations

Development of a hybrid MPI-thread programming system called PCU.Inter-thread message passing, including non-blocking collectives.A novel, scalable termination detection technique for communication rounds.Hybrid parallel scalability to 16K cores on an IBM Blue Gene/Q. Many of the world's leading supercomputer architectures are a hybrid of shared memory and network-distributed memory. Such an architecture lends itself to a hybrid MPI-thread programming model. We first present an implementation of inter-thread message passing based on the MPI and pthread libraries. In addition, we present an efficient implementation of termination detection for communication rounds. We use the term phased message passing to denote the communication interface based on this termination detection. This interface is then used to implement parallel operations for adaptive unstructured meshes, and the performance of resulting applications is compared to pure MPI operation. We also present new workflows enabled by the ability to vary the number of threads during runtime.