Exascale Computing Research Articles

Exascale Computing and Its Impact on Multiphysics Simulations: Current Status and Future Directions

This talk provides a detailed analysis of the current landscape and future directions in high-performance computing (HPC) systems, with a particular focus on the impact of evolving hardware trends on complex, multiphysics simulations. The recent deployment of the Frontier Supercomputer at Oak Ridge National Laboratory marked the beginning of the exascale computing era, closely followed by the introduction of the Aurora system at Argonne National Laboratory. Their deployment enabled the successful completion of the U.S. Department of Energy’s Exascale Computing Project (ECP)—a complementary, high-risk effort in re-designing software and scientific applications to take efficient use of these systems. Both Frontier and Aurora utilize specialized accelerators to achieve the power performance necessary for practical exascale deployment. This shift toward GPU-based systems has strongly influenced the HPC community’s approach to algorithms, impacting everything from implementation specifics to numerical strategies and mathematical modeling. Looking ahead, further specialization is inevitable, driven largely by the need for energy efficient neural network training and inference. This talk addresses the potential impact of these trends on multiphysics simulations, discussing both conventional and more innovative approaches to progress.

ExaPRR: A Framework for Support Dynamic and Interactive Events on Distributed Published Resource Repositories Mechanism in Distributed Exascale Computing Systems

This paper introduces the ExaPRR Framework, which is capable of managing and controlling dynamic and interactive events in addition to activities related to distributed Published Resources Repository, managing and controlling dynamic and interactive events of the said resource discovery (RD) by redefining the concept of request and response based on wave model. For this purpose, while analyzing the function of the Distributed Published Resource Repository RD and defining the concept of the next element participating in the global activity in this RD, it redefines the concept of request in the Distributed Exascale systems. This framework uses the concept of revising the state of implementation of dynamic and interactive events in the system and redefining the concept of request at every moment of the process of implementation of resource discovery activities based on system status, demanding requests, and dynamic and interactive event, manages the resource's discovery activities.

Accelerated kinetic model for global macro stability studies of high-beta fusion reactors

The field reversed configuration (FRC), such as studied in the C-2W experiment at TAE Technologies, is an attractive candidate for realizing a nuclear fusion reactor. In an FRC, kinetic ion effects play the majority role in macroscopic stability, which allows global stability studies to make use of fluid-kinetic hybrid (also referred to as Ohm's law) models wherein ions are treated kinetically while electrons are treated as a fluid. The development and validation of such a hybrid particle-in-cell algorithm in the Exascale Computing Project code WarpX are reported here. Implementation of this model in the WarpX framework benefits from the numerical efficiency of WarpX as well as its scalability on large HPC systems and portability to different architectures. Performance benchmarks of the new algorithm for large, 3-dimensional, full device simulations from the Perlmutter supercomputer are presented. Results of a series of FRC simulations are discussed in which the impact of two-fluid effects on the tilt-mode growth rate was studied. It was observed that, in agreement with previous Hall-MHD studies, two-fluid effects have a stabilizing impact on the tilt mode.

An Architecture for a Tri-Programming Model-Based Parallel Hybrid Testing Tool

As the development of high-performance computing (HPC) is growing, exascale computing is on the horizon. Therefore, it is imperative to develop parallel systems, such as graphics processing units (GPUs) and programming models, that can effectively utilise the powerful processing resources of exascale computing. A tri-level programming model comprising message passing interface (MPI), compute unified device architecture (CUDA), and open multi-processing (OpenMP) models may significantly enhance the parallelism, performance, productivity, and programmability of the heterogeneous architecture. However, the use of multiple programming models often leads to unexpected errors and behaviours during run-time. It is also difficult to detect such errors in high-level parallel programming languages. Therefore, this present study proposes a parallel hybrid testing tool that employs both static and dynamic testing techniques to address this issue. The proposed tool was designed to identify the run-time errors of C++ and MPI + OpenMP + CUDA systems by analysing the source code during run-time, thereby optimising the testing process and ensuring comprehensive error detection. The proposed tool was able to identify and categorise the run-time errors of tri-level programming models. This highlights the need for a parallel testing tool that is specifically designed for tri-level MPI + OpenMP + CUDA programming models. As contemporary parallel testing tools cannot, at present, be used to test software applications produced using tri-level MPI + OpenMP + CUDA programming models, this present study proposes the architecture of a parallel testing tool to detect run-time errors in tri-level MPI + OpenMP + CUDA programming models.

Identification of Multiscale Vortex Structures and Transition Fluid Features in CFD and Numerical Astrophysics

The need for effective postprocessor analysis of the results of extensive calculations on supercomputers remains extremely high when using new complex computing systems in modeling super- and hypersonic gas flows, predictive modeling of heat and mass transfer processes in jet engines and new energy devices, in solving multiscale problems for astrophysical simulation. Requirements for improving the quality and versatility of new software and visualization codes are increasing with the transition to exascale computing, which requires the improvement of new service systems, in particular systems for visualizing and analyzing the results of multi-scale calculations with dramatically increased input and output data. This paper considers the experience of using an authorized visualization system in the practice of solving some tasks with vortex flow structures and unpredictable instability.

Advances in Computational Approaches for Estimating Passive Permeability in Drug Discovery.

Passive permeation of cellular membranes is a key feature of many therapeutics. The relevance of passive permeability spans all biological systems as they all employ biomembranes for compartmentalization. A variety of computational techniques are currently utilized and under active development to facilitate the characterization of passive permeability. These methods include lipophilicity relations, molecular dynamics simulations, and machine learning, which vary in accuracy, complexity, and computational cost. This review briefly introduces the underlying theories, such as the prominent inhomogeneous solubility diffusion model, and covers a number of recent applications. Various machine-learning applications, which have demonstrated good potential for high-volume, data-driven permeability predictions, are also discussed. Due to the confluence of novel computational methods and next-generation exascale computers, we anticipate an exciting future for computationally driven permeability predictions.

GRaM-X: a new GPU-accelerated dynamical spacetime GRMHD code for Exascale computing with the Einstein Toolkit

We present GRaM-X (General Relativistic accelerated Magnetohydrodynamics on AMReX), a new GPU-accelerated dynamical-spacetime general relativistic magnetohydrodynamics (GRMHD) code which extends the GRMHD capability of Einstein Toolkit to GPU-based exascale systems. GRaM-X supports 3D adaptive mesh refinement (AMR) on GPUs via a new AMR driver for the Einstein Toolkit called CarpetX which in turn leverages AMReX, an AMR library developed for use by the United States DOE’s Exascale Computing Project. We use the Z4c formalism to evolve the Einstein equations and the Valencia formulation to evolve the equations of GRMHD. GRaM-X supports both analytic as well as tabulated equations of state. We implement TVD and WENO reconstruction methods as well as the HLLE Riemann solver. We test the accuracy of the code using a range of tests on static spacetime, e.g. 1D magnetohydrodynamics shocktubes, the 2D magnetic rotor and a cylindrical explosion, as well as on dynamical spacetimes, i.e. the oscillations of a 3D Tolman-Oppenheimer-Volkhof star. We find excellent agreement with analytic results and results of other codes reported in literature. We also perform scaling tests and find that GRaM-X shows a weak scaling efficiency of ∼40%–50% on 2304 nodes (13824 NVIDIA V100 GPUs) with respect to single-node performance on OLCF’s supercomputer Summit.

Challenges of high-fidelity virtual reactor for exascale computing and research progress of China Virtual Reactor

The virtual reactor is a sophisticated software system deployed on supercomputers to simulate physics process or predict component behaviors inside real reactors. It highly relies on advanced modeling and coupling methods, high performance computing and advanced verification and validation (V&V) techniques, which is another important method for reactor engineering design and mechanism research besides experiment. With the advent of exascale computer, it becomes mainstream trend to develop full-core high-fidelity simulation research under the help of exascale computing technology. This paper first summarizes the challenges of developing high-fidelity virtual reactor simulation in exascale computing. The challenges of high-fidelity virtual reactor simulation are introduced from four aspects: science-based modeling methods, high-resolution mesh model, scientific numerical methods , and multiphysics and multiscale coupling. Based on the architecture of exascale computer, the challenges of parallel optimization of virtual reactor for exascale computing are introduced. Furthermore, this paper introduces the recent research progress of China Virtual Reactor (CVR), which is about the parallel optimization for Chinese exascale computers and the functional extensions for achieving full-core high-fidelity simulation. The recent research progress of CVR includes multiscale simulation software of materials, fuel performance simulation software, thermal-hydraulic simulation software, full-core high-resolution meshing tool, and optimization of basic sparse operators.

Roadmap on electronic structure codes in the exascale era

Abstract Electronic structure calculations have been instrumental in providing many important insights into a range of physical and chemical properties of various molecular and solid-state systems. Their importance to various fields, including materials science, chemical sciences, computational chemistry, and device physics, is underscored by the large fraction of available public supercomputing resources devoted to these calculations. As we enter the exascale era, exciting new opportunities to increase simulation numbers, sizes, and accuracies present themselves. In order to realize these promises, the community of electronic structure software developers will however first have to tackle a number of challenges pertaining to the efficient use of new architectures that will rely heavily on massive parallelism and hardware accelerators. This roadmap provides a broad overview of the state-of-the-art in electronic structure calculations and of the various new directions being pursued by the community. It covers 14 electronic structure codes, presenting their current status, their development priorities over the next five years, and their plans towards tackling the challenges and leveraging the opportunities presented by the advent of exascale computing.

Parallel distributed productivity‐aware tree‐search using Chapel

AbstractWith the recent arrival of the exascale era, modern supercomputers are increasingly big making their programming much more complex. In addition to performance, software productivity is a major concern to choose a programming language, such as Chapel, designed for exascale computing. In this paper, we investigate the design of a parallel distributed tree‐search algorithm, namely P3D‐DFS, and its implementation using Chapel. The design is based on the Chapel's DistBag data structure, revisited by: (1) redefining the data structure for Depth‐First tree‐Search (DFS), henceforth renamed DistBag‐DFS; (2) redesigning the underlying load balancing mechanism. In addition, we propose two instantiations of P3D‐DFS considering the Branch‐and‐Bound (B&B) and Unbalanced Tree Search (UTS) algorithms. In order to evaluate how much performance is traded for productivity, we compare the Chapel‐based implementations of B&B and UTS to their best‐known counterparts based on traditional OpenMP (intra‐node) and MPI+X (inter‐node). For experimental validation using 4096 processing cores, we consider the permutation flow‐shop scheduling problem for B&B and synthetic literature benchmarks for UTS. The reported results show that P3D‐DFS competes with its OpenMP baselines for coarser‐grained shared‐memory scenarios, and with its MPI+X counterparts for distributed‐memory settings, considering both performance and productivity‐awareness. In the context of this work, this makes Chapel an alternative to OpenMP/MPI+X for exascale programming.

Near-continuum, hypersonic oxygen flow over a double cone simulated by direct simulation Monte Carlo informed from quantum chemistry

A large-scale, fully resolved direct simulation Monte Carlo (DSMC) computation of a non-equilibrium, reactive flow of pure oxygen over a double cone is presented. Under the simulated near-continuum conditions, the computational demands are shown to be significant because of the wide range of length scales that must be resolved. Therefore, robust grid adaption capabilities and efficient parallelization of the Stochastic PArallel Rarefied-gas Time-accurate Analyzer (SPARTA) code that is utilized in this work are essential. The thermochemical and transport collision models were selected for efficiency and simplicity. First-principles data, obtained from the highly accurate direct molecular simulation method, were used to inform the collision models’ parameters. Importantly, because SPARTA implements molecular collision models using collision-specific energies, the resulting macroscopic relaxation rates were evaluateda posteriorivia zero-dimensional heat bath simulations. The comparisons of surface properties, namely heat flux and pressure, show very close agreement with previous computational fluid dynamics (CFD) results. Differences with the measurements were found to be similar to the CFD simulations. The unresolved discrepancy with the measurements could be due to inconsistent free stream conditions with the actual experimental data or missing physical phenomena altogether, for example atomic and molecular oxygen electronically excited states, three-dimensional effects, or more complex gas–surface interactions. As shown in this work, the advantages of obtaining a DSMC particle solution for these flows reside in the method's ability to be directly informed from first principles and to seamlessly describe internal energy non-equilibrium for all modes. With the advent of exascale computing and beyond, particle methods will be an increasingly important tool to verify the validity of physical assumptions in reduced-order models via fully resolved, experimental-scale simulations, down to the level of molecular-level distributions.

Exascale Computing Project's Broadening Participation Initiative

Exascale Computing Project's Broadening Participation Initiative

Redesigning OpenKMC for Multi-Component Trillion-Atom Simulations on the New Sunway Supercomputer

The atomic kinetic Monte Carlo method plays an important role in material simulations by connecting the microscale mechanism with macroscale evolution. However, the long-time simulation of multi-component materials is highly challenging because it demands significant computing resources. With the advent of exascale computing, ultra-high computing power can enable kinetic Monte Carlo (KMC) simulations. In this paper, we deeply optimize OpenKMC for the new-generation Sunway supercomputer. This includes optimizing the memory access for the SW39000 architecture, eliminating various redundant computations at growing scales, and proposing a communication strategy for heterogeneous platforms. In addition, we expanded OpenKMC's simulation for multi-component alloys. Finally, the acceleration framework can produces a <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$37\times$</tex-math></inline-formula> performance enhancement on the Sunway platform. Furthermore, when powered by 10 million cores, our program can perform trillion-atom simulations of complex multi-component alloys with 85% parallel efficiency.

PluginPlay: Enabling exascale scientific software one module at a time.

For many computational chemistry packages, being able to efficiently and effectively scale across an exascale cluster is a heroic feat. Collective experience from the Department of Energy's Exascale Computing Project suggests that achieving exascale performance requires far more planning, design, and optimization than scaling to petascale. In many cases, entire rewrites of software are necessary to address fundamental algorithmic bottlenecks. This in turn requires a tremendous amount of resources and development time, resources that cannot reasonably be afforded by every computational science project. It thus becomes imperative that computational science transition to a more sustainable paradigm. Key to such a paradigm is modular software. While the importance of modular software is widely recognized, what is perhaps not so widely appreciated is the effort still required to leverage modular software in a sustainable manner. The present manuscript introduces PluginPlay, https://github.com/NWChemEx-Project/PluginPlay, an inversion-of-control framework designed to facilitate developing, maintaining, and sustaining modular scientific software packages. This manuscript focuses on the design aspects of PluginPlay and how they specifically influence the performance of the resulting package. Although, PluginPlay serves as the framework for the NWChemEx package, PluginPlay is not tied to NWChemEx or even computational chemistry. We thus anticipate PluginPlay to prove to be a generally useful tool for a number of computational science packages looking to transition to the exascale.

Solving the Neutron Transport Equation for Microreactor Modeling Using Unstructured Meshes and Exascale Computing Architectures

The Microreactor Exascale eZ CALculation (MEZCAL) tool has been developed to accurately and efficiently solve the neutron transport equation in general, unstructured meshes to support the design and modeling of microreactors. MEZCAL solves the self-adjoint angular flux form of the neutron transport equation using the finite element method. As the neutron transport equation is computationally expensive to solve, MEZCAL is designed to efficiently use exascale computing architectures, with an emphasis on graphics processing unit computing. To leverage existing tools, MEZCAL is built using the MFEM library and uses solvers from HYPRE, PETSc, and SLEPc. Verification of the neutron transport solver in MEZCAL is demonstrated with the solution to a one-dimensional cylindrical problem that has a semi-analytic solution. After verification, a realistic microreactor based on the MARVEL microreactor design is modeled using MEZCAL. Spatial and angular refinement results are presented for a two-dimensional model of the MARVEL microreactor, and the eigenvalue is converged to approximately 60 pcm. This convergence required a very fine mesh and more than 3.76 Billion Degrees Of Freedom (BDOF). Preliminary results are also presented for a three-dimensional model of the MARVEL microreactor. Finally, a weak scaling study is performed to investigate how the methods in MEZCAL will scale for larger problems with the next generation of exascale computing architectures.

ANKH: A Generalized O(N) Interpolated Ewald Strategy for Molecular Dynamics Simulations

To evaluate electrostatics interactions, molecular dynamics (MD) simulations rely on Particle Mesh Ewald (PME), an algorithm that uses Fast Fourier Transforms (FFTs) or, alternatively, on Fast Multipole Methods (FMM) approaches. However, the FFTs low scalability remains a strong bottleneck for large-scale PME simulations on supercomputers. On the opposite, FFT-free FMM techniques are able to deal efficiently with such systems but they fail to reach PME performances for small- to medium-size systems, limiting their real-life applicability. We propose ANKH, a strategy grounded on interpolated Ewald summations and designed to remain efficient/scalable for any size of systems. The method is generalized for distributed point multipoles, and so for induced dipoles, which makes it suitable for high performance simulations using new generation polarizable force fields toward exascale computing.

An Extensive Power and Performance Analysis for High Dimensional Mesh and Torus Interconnection Networks

The next generation parallel computers are the keys to achieve exascale performance, whereas sequential computers have already been saturated. In order to achieve this mighty target of exascale computing, one of the main challenges is the reduction of power consumption along with achieving suitable performance. Energy efficiency is a key feature to ensure the trade-off between the performances over the required power usages. Hence, to focus on those issues, the target of this article is to analyze the performance versus power usage trade-off for the conventional networks like- Mesh and Torus. High degree networks show much better performance than the low degree of networks. However, high degree networks require higher power usage for their high degree of interconnected links. This article showed that with zero load latency, the 3D Torus could show about 57.07% better performance than a 2D Torus. On the other hand, a 2D Mesh network requires about 24.22% less router power usage than the 3D Mesh, &amp; 5D Torus requires about 66.8% higher router power usage than a 3D Torus network.

The Applications of the Quantum Computer and the Exascale Computer [Commentary] [Commentary

Supercomputers are making rapid progress. For the world's fastest supercomputer “Frontier,” the computing speed has reached 10 trillion floating-point instruction cycles per second (10 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">18</sup> , exaFLOPS), making many existing complex problems, including weather forecasting and disaster prevention, more accurate than ever. However, the calculation of real systems with a large number of electrons is still difficult. Exascale computers can simulate some systems with dual quantum states, but in the case of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</i> dual quantum states, the exponential growth of 2 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</sup> makes the calculation to simulate real system impossible. If we want to understand the dynamic quantum state evolution of a real matter, it is even more difficult to track the dynamics of “every possible” configuration of a huge number of 2 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</sup> quantum states. If n is 20, it will take about 128 KB of memory to store with the current computer's memory, which can be done by ordinary computers. When there are hundreds of electrons, the memory required to store this system exceeds the number of particles in the entire universe. Unfortunately, almost all materials and drugs have at least hundreds of electrons, which means that direct use of Exascale computers to develop and design any new drugs and materials also becomes impossible. At the same time, after the popularization of artificial intelligence and deep learning, it is also found that the computing power, bandwidth, and memory space of classical computers are not enough, while quantum computing has at least the following three advantages and has the opportunity to provide solutions for complex systems.

Optimization Techniques for GPU Programming

In the past decade, Graphics Processing Units have played an important role in the field of high-performance computing and they still advance new fields such as IoT, autonomous vehicles, and exascale computing. It is therefore important to understand how to extract performance from these processors, something that is not trivial. This survey discusses various optimization techniques found in 450 articles published in the last 14 years. We analyze the optimizations from different perspectives which shows that the various optimizations are highly interrelated, explaining the need for techniques such as auto-tuning.

Simulations in the era of exascale computing.

Exascale computers - supercomputers that can perform 1018 floating point operations per second-started coming online in 2022: in the United States, Frontier launched as the first public exascale supercomputer and Aurora is due to open soon; OceanLight and Tianhe-3 are operational in China; and JUPITER is due to launch in 2023 in Europe. Supercomputers offer unprecedented opportunitiesfor modelling complex materials. In this Viewpoint, five researchers working on different types of materials discuss the most promising directions in computational materials science.