Exascale Computing Research Articles

The PetscSF Scalable Communication Layer

PetscSF, the communication component of the Portable, Extensible Toolkit for Scientific Computation (PETSc), is designed to provide PETSc's communication infrastructure suitable for exascale computers that utilize GPUs and other accelerators. PetscSF provides a simple application programming interface (API) for managing common communication patterns in scientific computations by using a star-forest graph representation. PetscSF supports several implementations based on MPI and NVSHMEM, whose selection is based on the characteristics of the application or the target architecture. An efficient and portable model for network and intra-node communication is essential for implementing large-scale applications. The Message Passing Interface, which has been the de facto standard for distributed memory systems, has developed into a large complex API that does not yet provide high performance on the emerging heterogeneous CPU-GPU-based exascale systems. In this article, we discuss the design of PetscSF, how it can overcome some difficulties of working directly with MPI on GPUs, and we demonstrate its performance, scalability, and novel features.

Accelerating Geostatistical Modeling and Prediction With Mixed-Precision Computations: A High-Productivity Approach With PaRSEC

Geostatistical modeling, one of the prime motivating applications for exascale computing, is a technique for predicting desired quantities from geographically distributed data, based on statistical models and optimization of parameters. Spatial data are assumed to possess properties of stationarity or non-stationarity via a kernel fitted to a covariance matrix. A primary workhorse of stationary spatial statistics is Gaussian maximum log-likelihood estimation (MLE), whose central data structure is a dense, symmetric positive definite covariance matrix of the dimension of the number of correlated observations. Two essential operations in MLE are the application of the inverse and evaluation of the determinant of the covariance matrix. These can be rendered through the Cholesky decomposition and triangular solution. In this contribution, we reduce the precision of weakly correlated locations to single- or half- precision based on distance. We thus exploit mathematical structure to migrate MLE to a three-precision approximation that takes advantage of contemporary architectures offering BLAS3-like operations in a single instruction that are extremely fast for reduced precision. We illustrate application-expected accuracy worthy of double-precision from a majority half-precision computation, in a context where uniform single-precision is by itself insufficient. In tackling the complexity and imbalance caused by the mixing of three precisions, we deploy the <monospace>PaRSEC</monospace> runtime system. <monospace>PaRSEC</monospace> delivers on-demand casting of precisions while orchestrating tasks and data movement in a multi-GPU distributed-memory environment within a tile-based Cholesky factorization. Application-expected accuracy is maintained while achieving up to <inline-formula><tex-math notation="LaTeX">$1.59X$</tex-math></inline-formula> by mixing FP64/FP32 operations on 1536 nodes of <monospace>HAWK</monospace> or 4096 nodes of <monospace>Shaheen II</monospace> , and up to <inline-formula><tex-math notation="LaTeX">$2.64X$</tex-math></inline-formula> by mixing FP64/FP32/FP16 operations on 128 nodes of <monospace>Summit</monospace> , relative to FP64-only operations. This translates into up to 4.5, 4.7, and 9.1 (mixed) PFlop/s sustained performance, respectively, demonstrating a synergistic combination of exascale architecture, dynamic runtime software, and algorithmic adaptation applied to challenging environmental problems.

Demonstrating the viability of Lagrangian in situ reduction on supercomputers

Performing exploratory analysis and visualization of large-scale time-varying computational science applications is challenging due to inaccuracies that arise from under-resolved data. In recent years, Lagrangian representations of the vector field computed using in situ processing are being increasingly researched and have emerged as a potential solution to enable exploration. However, prior works have offered limited estimates of the encumbrance on the simulation code as they consider “theoretical” in situ environments. Further, the effectiveness of this approach varies based on the nature of the vector field, benefitting from an in-depth investigation for each application area. With this study, an extended version of Sane et al. (2021), we contribute an evaluation of Lagrangian analysis viability and efficacy for simulation codes executing at scale on a supercomputer. We investigated previously unexplored cosmology and seismology applications as well as conducted a performance benchmarking study by using a hydrodynamics mini-application targeting exascale computing. To inform encumbrance, we integrated in situ infrastructure with simulation codes, and evaluated Lagrangian in situ reduction in representative homogeneous and heterogeneous HPC environments. To inform post hoc accuracy, we conducted a statistical analysis across a range of spatiotemporal configurations as well as a qualitative evaluation. Additionally, our study contributes cost estimates for distributed-memory post hoc reconstruction. In all, we demonstrate viability for each application — data reduction to less than 1% of the total data via Lagrangian representations, while maintaining accurate reconstruction and requiring under 10% of total execution time in over 90% of our experiments.

Magnetic reconnection in the era of exascale computing and multiscale experiments

Astrophysical plasmas have the remarkable ability to preserve magnetic topology, which inevitably gives rise to the accumulation of magnetic energy within stressed regions including current sheets. This stored energy is often released explosively through the process of magnetic reconnection, which produces a reconfiguration of the magnetic field, along with high-speed flows, thermal heating and nonthermal particle acceleration. Either collisional or kinetic dissipation mechanisms are required to overcome the topological constraints, both of which have been predicted by theory and validated with in situ spacecraft observations or laboratory experiments. However, major challenges remain in understanding magnetic reconnection in large systems, such as the solar corona, where the collisionality is weak and the kinetic scales are vanishingly small in comparison with macroscopic scales. The plasmoid instability or formation of multiple plasmoids in long, reconnecting current sheets is one possible multiscale solution for bridging this vast range of scales, and new laboratory experiments are poised to study these regimes. In conjunction with these efforts, we anticipate that the coming era of exascale computing, together with the next generation of observational capabilities, will enable new progress on a range of challenging problems, including the energy build-up and onset of reconnection, partially ionized regimes, the influence of magnetic turbulence and particle acceleration.

A massively parallel time-domain coupled electrodynamics–micromagnetics solver

We present a high-performance coupled electrodynamics–micromagnetics solver for full physical modeling of signals in microelectronic circuitry. The overall strategy couples a finite-difference time-domain approach for Maxwell’s equations to a magnetization model described by the Landau–Lifshitz–Gilbert equation. The algorithm is implemented in the Exascale Computing Project software framework, AMReX, which provides effective scalability on manycore and GPU-based supercomputing architectures. Furthermore, the code leverages ongoing developments of the Exascale Application Code, WarpX, which is primarily being developed for plasma wakefield accelerator modeling. Our temporal coupling scheme provides second-order accuracy in space and time by combining the integration steps for the magnetic field and magnetization into an iterative sub-step that includes a trapezoidal temporal discretization for the magnetization. The performance of the algorithm is demonstrated by the excellent scaling results on NERSC multicore and GPU systems, with a significant (59×) speedup on the GPU using a node-by-node comparison. We demonstrate the utility of our code by performing simulations of an electromagnetic waveguide and a magnetically tunable filter.

ExCALIBUR—U.K.’s Preparation for the Arrival of the Next Generation of HPC

The articles in this special section focus on the ExCalibur high performance computing program. In 2019, the UK Government funded the Exascale Computing Algorithms and Infrastructures Benefitting UK Research (ExCALIBUR) Programa with an investment of £45.7m, led by the Met Office and EPSRC along with other U.K. research bodies. The vision of ExCALIBUR is to prepare the U.K. hardware and software landscape for the emerging age of exascale computing. The ExCALIBUR priorities are as follows: 1) Separation of Concerns, 2) Co-Design, 3) Data Science, and 4) Investing in People. In 2020, in the first round of ExCALIBUR, 10 software Design and Development Working Groups (DDWG) were formed, each representing a different community and tasked with developing high-priority use cases, the cost-effective strategy to enable these, and demonstrate collaborative progress toward the successful transition to exascale. Significant funds have been invested in 2021 to support a number of these DDWGs for a further three years with some of the DDWGs combining their efforts after identifying common possible solutions to remove underlying bottlenecks in their different applications. These longer three-year projects will see substantial software and other outputs, benefiting the international HPC community. In terms of hardware, the U.K

Solving the Neutron Transport Equation for Microreactor Modeling Using Unstructured Meshes and Exascale Computing Architectures

Solving the Neutron Transport Equation for Microreactor Modeling Using Unstructured Meshes and Exascale Computing Architectures

Empirical optimization of molecular simulation force fields by Bayesian inference

The demands on the accuracy of force fields for classical molecular dynamics simulations are steadily growing as larger and more complex systems are studied over longer times. One way to meet these growing demands is to hand over the learning of force fields and their parameters to machines in a systematic (semi)automatic manner. Doing so, we can take full advantage of exascale computing, the increasing availability of experimental data, and advances in quantum mechanical computations and the calculation of experimental observables from molecular ensembles. Here, we discuss and illustrate the challenges one faces in this endeavor and explore a way forward by adapting the Bayesian inference of ensembles (BioEn) method [Hummer and Köfinger, J. Chem. Phys. (2015)] for force field parameterization. In the Bayesian inference of force fields (BioFF) method developed here, the optimization problem is regularized by a simplified prior on the force field parameters and an entropic prior acting on the ensemble. The latter compensates for the unavoidable over simplifications in the parameter prior. We determine optimal force field parameters using an iterative predictor–corrector approach, in which we run simulations, determine the reference ensemble using the weighted histogram analysis method (WHAM), and update the force field according to the BioFF posterior. We illustrate this approach for a simple polymer model, using the distance between two labeled sites as the experimental observable. By systematically resolving force field issues, instead of just reweighting a structural ensemble, the BioFF corrections extend to observables not included in ensemble reweighting. We envision future force field optimization as a formalized, systematic, and (semi)automatic machine-learning effort that incorporates a wide range of data from experiment and high-level quantum chemical calculations, and takes advantage of exascale computing resources.Graphic abstract

Technology for Supercomputer Simulation of Turbulent Flows in the Good New Days of Exascale Computing

A technology for scale-resolving simulations of turbulent flows in the problems of aerodynamics and aeroacoustics is presented. It is based on the higher accuracy numerical schemes on unstructured mixed-element meshes and latest non-zonal hybrid approaches combining Reynoldsaveraged Navier – Stokes (RANS) and Large eddy simulation (LES) methods for turbulence modeling. It targets a wide range of high performance computing (HPC) systems, from a compute server or small cluster to an exascale supercomputer. The advantages of the key components of the technology are summarized. These key components are a hybrid RANS-LES turbulence modeling method, a numerical scheme for discretization in space, a parallel algorithm, and a portable software implementation for modern hybrid systems with extra massive parallelism. Examples of our simulations are given and parallel performance on various HPC systems is presented.

Co-design in the Exascale Computing Project

We provide an overview of the six co-design centers within the U.S. Department of Energy’s Exascale Computing Project, each of which is described in more detail in a separate paper in this special issue. We also give a perspective on the evolution of computational co-design.

Introduction to the Department of Energy Leadership Computing Facilities

Did you ever say to yourself "Boy I wished I had a bigger computer." Do you look at your computation and see that cloud computing is not a viable solution? Do you think your problem could benefit from *really* high performance computing but don't know where to start in trying to access and use it? Acoustics researchers typically fail to think "big" with respect to their simulations—finding ways to reduce and limit their problems to fit a single workstation, a small cluster, a small HPC, or the cloud. The Department of Energy (DOE) Leadership Computing Facilities (LCF) are a set of the highest performance publicly accessible computing facilities in the U.S. Summit at Oak Ridge National Laboratory and Perlmutter at Lawrence Berkeley National Laboratory are #2 and #5 of the Top500 HPC list. Two Exascale computers: Frontier at Oak Ridge and Aurora at Argonne National Laboratory are set to arrive in the next couple of years. This presentation will introduce the DOE LCF and several of the allocation programs by which researchers and industry can gain access and training to use these incredible facilities.

Co-design Center for Exascale Machine Learning Technologies (ExaLearn)

Rapid growth in data, computational methods, and computing power is driving a remarkable revolution in what variously is termed machine learning (ML), statistical learning, computational learning, and artificial intelligence. In addition to highly visible successes in machine-based natural language translation, playing the game Go, and self-driving cars, these new technologies also have profound implications for computational and experimental science and engineering, as well as for the exascale computing systems that the Department of Energy (DOE) is developing to support those disciplines. Not only do these learning technologies open up exciting opportunities for scientific discovery on exascale systems, they also appear poised to have important implications for the design and use of exascale computers themselves, including high-performance computing (HPC) for ML and ML for HPC. The overarching goal of the ExaLearn co-design project is to provide exascale ML software for use by Exascale Computing Project (ECP) applications, other ECP co-design centers, and DOE experimental facilities and leadership class computing facilities.

GPU algorithms for Efficient Exascale Discretizations

In this paper we describe the research and development activities in the Center for Efficient Exascale Discretization within the US Exascale Computing Project, targeting state-of-the-art high-order finite-element algorithms for high-order applications on GPU-accelerated platforms. We discuss the GPU developments in several components of the CEED software stack, including the libCEED, MAGMA, MFEM, libParanumal, and Nek projects. We report performance and capability improvements in several CEED-enabled applications on both NVIDIA and AMD GPU systems.

Towards performance portability in the Spark astrophysical magnetohydrodynamics solver in the Flash-X simulation framework

Simulations of core-collapse supernovae, and other astrophysical phenomena, are quintessential extreme-scale computing challenges. For core-collapse supernova simulations to be carried out by the ExaStar project under the Exascale Computing Project umbrella, a robust, efficient, and state-of-the-art magnetohydrodynamics solver is a critical requirement. In Flash-X, the primary software instrument for ExaStar, a new magnetohydrodynamics solver has been designed and implemented from the ground up to achieve accuracy and efficiency for simulations of complex astrophysical flows. This new solver, dubbed Spark, uses high-order spatial reconstruction, Runge–Kutta time integration, and an efficient cell-centered approach to satisfying the divergence-free condition for the magnetic fields. Spark was written to be optimized for data locality in cache hierarchy of CPUs. Since data locality optimizations for cache hierarchy are not directly compatible with those of accelerators, we have taken the approach of using program synthesis to avoid massive amounts of code replication that would be necessary if we were to maintain two different versions of the solver. Our program synthesis relies on a simple key-dictionary approach, implemented in python, that enables us to assemble the version of the solver suitable for the target hardware from code fragments identified by specific keys. In this paper, we describe the data locality optimizations of the solver for CPUs and accelerators and the program synthesis tools that enable this portability. We also detail the parallel performance of Spark for both CPUs and accelerators.

Measurement and analysis of GPU-accelerated applications with HPCToolkit

To address the challenge of performance analysis on the US DOE’s forthcoming exascale supercomputers, Rice University has been extending its HPCToolkit performance tools to support measurement and analysis of GPU-accelerated applications. To help developers understand the performance of accelerated applications as a whole, HPCToolkit’s measurement and analysis tools attribute metrics to calling contexts that span both CPUs and GPUs. To measure GPU-accelerated applications efficiently, HPCToolkit employs a novel wait-free data structure to coordinate monitoring and attribution of GPU performance. To help developers understand the performance of complex GPU code generated from high-level programming models, HPCToolkit constructs sophisticated approximations of call path profiles for GPU computations. To support fine-grained analysis and tuning, HPCToolkit uses PC sampling and instrumentation to measure and attribute GPU performance metrics to source lines, loops, and inlined code. To supplement fine-grained measurements, HPCToolkit can measure GPU kernel executions using hardware performance counters. To provide a view of how an execution evolves over time, HPCToolkit can collect, analyze, and visualize call path traces within and across nodes. Finally, on NVIDIA GPUs, HPCToolkit can derive and attribute a collection of useful performance metrics based on measurements using GPU PC samples. We illustrate HPCToolkit’s new capabilities for analyzing GPU-accelerated applications with several codes developed as part of the Exascale Computing Project.

Enabling GPU accelerated computing in the SUNDIALS time integration library

As part of the Exascale Computing Project (ECP), a recent focus of development efforts for the SUite of Nonlinear and DIfferential/ALgebraic equation Solvers (SUNDIALS) has been to enable GPU-accelerated time integration in scientific applications at extreme scales. This effort has resulted in several new GPU-enabled implementations of core SUNDIALS data structures, support for programming paradigms which are aware of the heterogeneous architectures, and the introduction of utilities to provide new points of flexibility. In this paper, we discuss our considerations, both internal and external, when designing these new features and present the features themselves. We also present performance results for several of the features on the Summit supercomputer and early access hardware for the Frontier supercomputer, which demonstrate negligible performance overhead resulting from the additional infrastructure and significant speedups when using both NVIDIA and AMD GPUs.

Posits and the state of numerical representations in the age of exascale and edge computing

AbstractGrowing constraints on memory utilization, power consumption, and I/O throughput have increasingly become limiting factors to the advancement of high performance computing (HPC) and edge computing applications. IEEE‐754 floating‐point types have been the de facto standard for floating‐point number systems for decades, but the drawbacks of this numerical representation leave much to be desired. Alternative representations are gaining traction, both in HPC and machine learning environments. Posits have recently been proposed as a drop‐in replacement for the IEEE‐754 floating‐point representation. We survey the state‐of‐the‐art and state‐of‐the‐practice in the development and use of posits in edge computing and HPC. The current literature supports posits as a promising alternative to traditional floating‐point systems, both as a stand‐alone replacement and in a mixed‐precision environment. Development and standardization of the posit type is ongoing, and much research remains to explore the application of posits in different domains, how to best implement them in hardware, and where they fit with other numerical representations.

Performance Portability in the Exascale Computing Project: Exploration Through a Panel Series

Performance portability is a critical issue for the Exascale Computing Project (ECP) because of nontrivial architectural differences between machines available today and those expected at exascale. Many ECP project teams are working toward performance portability, and would expect to benefit from sharing lessons learned, identifying gaps, and discovering opportunities for partnerships. To facilitate this communication, the IDEAS-ECP project partnered with the three focus areas of ECP (application development, software technology, and hardware and integration), and Department of Energy computing facilities, to lead a series of panel discussions on performance portability. The panels were organized around broadly common themes of algorithmic and data locality challenges. In this article, we describe the panel series, its objectives, and perspectives from the various areas of the project. We also discuss use cases that are distinctive, as well as conclusions drawn from the collective experience of the participants.

Understanding the Role of Endothelial Glycocalyx in Mechanotransduction via Computational Simulation: A Mini Review.

Endothelial glycocalyx (EG) is a forest-like structure, covering the lumen side of blood vessel walls. EG is exposed to the mechanical forces of blood flow, mainly shear, and closely associated with vascular regulation, health, diseases, and therapies. One hallmark function of the EG is mechanotransduction, which means the EG senses the mechanical signals from the blood flow and then transmits the signals into the cells. Using numerical modelling methods or in silico experiments to investigate EG-related topics has gained increasing momentum in recent years, thanks to tremendous progress in supercomputing. Numerical modelling and simulation allows certain very specific or even extreme conditions to be fulfilled, which provides new insights and complements experimental observations. This mini review examines the application of numerical methods in EG-related studies, focusing on how computer simulation contributes to the understanding of EG as a mechanotransducer. The numerical methods covered in this review include macroscopic (i.e., continuum-based), mesoscopic [e.g., lattice Boltzmann method (LBM) and dissipative particle dynamics (DPD)] and microscopic [e.g., molecular dynamics (MD) and Monte Carlo (MC) methods]. Accounting for the emerging trends in artificial intelligence and the advent of exascale computing, the future of numerical simulation for EG-related problems is also contemplated.

Spatial specification of hypertorus interconnect by infinite and reenterable coloured Petri nets

Multidimensional torus interconnect finds wide application in modern exascale computing. For models design in high-performance computing, grid and cloud computing, and also systems biology, two basic ways of specifying spatial structures with Petri nets are considered – an infinite Petri net specified by a parametric expression (PE) and a reenterable coloured Petri net (CPN). The paper studies a composition of hypertorus grid models in the form of a PE and a reenterable CPN, their mutual transformations, and unfolding into a place/transition net; the parameters are the number of dimensions and the size of grid. A grid is composed via connection of neighbouring cells by dedicated transitions modelling channels. Reenterable model peculiarities are explained on step-by-step simulation examples. The rules of mutual transformations of Petri net spatial specifications are specified. Comparative investigation of two mentioned forms of spatial specifications is implemented, including analysis techniques and tools. CPNs are convenient for the state space analysis. The main advantage of PEs is the ability to obtain linear invariants and other structural constructs of Petri nets, for instance, siphons and traps, in parametric form that allows us to draw conclusions on Petri net properties for any values of parameters.