Petascale Supercomputers Research Articles

Simulations of plasmas and fluids using anti-symmetric models

ALMA (Anti-symmetric, Large-Moment, Accelerated) is a fast, flexible, and scalable toolkit designed to solve hyperbolic conservation law systems in hybrid supercomputers. This manuscript describes the theoretical background and implementation of ALMA, which uses the anti-symmetric formulation of fluids to obtain simple, robust, and easily paralellizable code. Practical GPU acceleration is realized on entire applications with an overall gain factor of 2 to 4. ALMA also provides a parallel, GPU accelerated sparse solver based on geometric multigrid, capable of diagonalizing linear systems with 239 unknowns. We demonstrate ALMA's scaling and performance in petascale supercomputers and use standard fluid models to verify the overall approach with canonical benchmark problems.

Toward Exascale: Overview of Large Eddy Simulations and Direct Numerical Simulations of Nuclear Reactor Flows with the Spectral Element Method in Nek5000

At the beginning of the last decade, Petascale supercomputers (i.e., computers capable of more than 1 petaFLOP) emerged. Now, at the dawn of exascale supercomputing, we provide a review of recent landmark simulations of portions of reactor components with turbulence-resolving techniques that this computational power has made possible. In fact, these simulations have provided invaluable insight into flow dynamics, which is difficult or often impossible to obtain with experiments alone. We focus on simulations performed with the spectral element method, as this method has emerged as a powerful tool to deliver massively parallel calculations at high fidelity by using large eddy simulation or direct numerical simulation. We also limit this paper to constant-property incompressible flow of a Newtonian fluid in the absence of other body or external forces, although the method is by no means limited to this class of flows. We briefly review the fundamentals of the method and the reasons it is compelling for the simulation of nuclear engineering flows. We review in detail a series of Petascale simulations, including the simulations of helical coil steam generators, fuel assemblies, and pebble beds. Even with Petascale computing, however, limitations for nuclear modeling and simulation tools remain. In particular, the size and scope of turbulence-resolving simulations are still limited by computing power and resolution requirements, which scale with the Reynolds number. In the final part of this paper, we discuss the future of the field, including recent advancements in emerging architectures such as GPU-based supercomputers, which are expected to power the next generation of high-performance computers.

ExaWind: A multifidelity modeling and simulation environment for wind energy

We introduce the open-source ExaWind modeling and simulation environment for wind energy. The primary physics codes of ExaWind are Nalu-Wind and OpenFAST. Nalu-Wind is a wind-focused computational fluid dynamics (CFD) code that is coupled to the whole-turbine simulation code OpenFAST. The ExaWind environment was created under U.S. Department of Energy funding to achieve the highest-fidelity simulations of wind turbines and wind farms to date, with the goal of enabling disruptive changes to turbine and plant design and operation. Innovation will be gleaned through better understanding of the complex flow dynamics in wind farms, including wake evolution and the impact of wakes on downstream turbines and turbulent flow from complex terrain. High-fidelity predictive simulations employ hybrid turbulence models, geometry/boundary-layer-resolving CFD meshes, atmospheric turbulence, nonlinear structural dynamics, and fluid-structure interaction. While there is an emphasis on very high-fidelity simulations (e.g., blade resolved with full fluid-structure coupling), the ExaWind environment supports lower-fidelity modeling capabilities including actuator-line and -disk methods. Important in the development of ExaWind codes is that the codes scale well on today’s largest petascale supercomputers and on the next-generation platforms that will enable exascale computing.

Adapting Computer Arithmetic Structures to Sustainable Supercomputing in Low-Power, Majority-Logic Nanotechnologies

Petascale supercomputers are already pushing power boundaries that can be supplied or dissipated cost-effectively; greater challenges await us in the era of exascale machines. We are thus motivated to study methods of reducing the energy cost of arithmetic operations, which can be substantial in numerically intensive applications. Additionally, being both a widely-used operation in itself and an important building block for synthesizing other arithmetic operations, has received much attention in this regard. Circuit and energy costs of fast adders are dominated by their fast carry networks. The availability of simple and energy-efficient majority function in certain emerging nanotechnologies (such as quantum-dot cellular automata, single-electron tunneling, tunneling phase logic, magnetic tunnel junction, nanoscale bar magnets, and memristors) has motivated our work to reformulate the carry recurrence in terms of fully-utilized majority elements, with all three inputs usefully employed. We compare our novel designs and resulting circuits to prior proposals based on 3-input majority elements in quantum-dot cellular automata, demonstrating advantages in both speed and circuit complexity. We also show that the performance and cost advantages carry over to at least one other emerging, energy-efficient technology, single-electron tunneling, raising hopes for achieving similar benefits with other technologies, which we review very briefly.

Flexibility of Free and AcrB-Bound AcrA in the AcrAB-TolC Multidrug Efflux Pump of Escherichia coli Determined using 3D PMFS

Key components of the emerging global epidemic of antibiotic resistance are the multidrug efflux pumps of pathogenic, Gram-negative bacteria, which span the periplasmic region between the inner and outer membranes of the cell. These protein complexes are an innate resistance mechanism, removing harmful antibiotics from bacteria. Until 2014 many aspects of the mechanisms for pump activity remained elusive due to the lack of structural data. Since then several increasingly detailed electron microscopy maps of an entire efflux pump complex, AcrAB-TolC, have been resolved, resulting in atomic-level structural models. Using these new models, we performed molecular dynamics simulations to study one of the key components of the protein complex, AcrA, which connects the inner-membrane-bound AcrB to the outer-membrane-bound TolC. We determined the flexibility of free AcrA by calculating a three-dimensional potential of mean force (PMF) focused on three angles that govern AcrA's conformational dynamics. AcrA shows a wide range of accessible orientations, with two main energy basins separated by a low (< 4 kT) barrier, consistent with previously reported equilibrium simulations. The conformation in one of the basins is similar to the AcrB-bound state, suggesting that assembly proceeds by conformational selection. Additional PMF calculations of AcrA conformation when bound to AcrB further elucidate how binding to TolC occurs. The 3D PMF calculations required petascale supercomputing resources and tens of microseconds in aggregate to carry out.

Extreme-Scale Block-Structured Adaptive Mesh Refinement

In this article, we present a novel approach for block-structured adaptive mesh refinement (AMR) that is suitable for extreme-scale parallelism. All data structures are designed such that the size of the meta data in each distributed processor memory remains bounded independent of the processor number. In all stages of the AMR process, we use only distributed algorithms. No central resources such as a master process or replicated data are employed, so that an unlimited scalability can be achieved. For the dynamic load balancing in particular, we propose to exploit the hierarchical nature of the block-structured domain partitioning by creating a lightweight, temporary copy of the core data structure. This copy acts as a local and fully distributed proxy data structure. It does not contain simulation data, but only provides topological information about the domain partitioning into blocks. Ultimately, this approach enables an inexpensive, local, diffusion-based dynamic load balancing scheme. We demonstrate the excellent performance and the full scalability of our new AMR implementation for two architecturally different petascale supercomputers. Benchmarks on an IBM Blue Gene/Q system with a mesh containing 3.7 trillion unknowns distributed to 458,752 processes confirm the applicability for future extreme-scale parallel machines. The algorithms proposed in this article operate on blocks that result from the domain partitioning. This concept and its realization support the storage of arbitrary data. In consequence, the software framework can be used for different simulation methods, including mesh based and meshless methods. In this article, we demonstrate fluid simulations based on the lattice Boltzmann method.

A Large-Scale Study of Failures on Petascale Supercomputers

With the rapid development of supercomputers, the scale and complexity are ever increasing, and the reliability and resilience are faced with larger challenges. There are many important technologies in fault tolerance, such as proactive failure avoidance technologies based on fault prediction, reactive fault tolerance based on checkpoint, and scheduling technologies to improve reliability. Both qualitative and quantitative descriptions on characteristics of system faults are very critical for these technologies. This study analyzes the source of failures on two typical petascale supercomputers called Sunway BlueLight (based on multi-core CPUs) and Sunway TaihuLight (based on heterogeneous manycore CPUs). It uncovers some interesting fault characteristics and finds unknown correlation relationship among main components’ faults. Finally the paper analyzes the failure time of the two supercomputers in various grains of resource and different time spans, and builds a uniform multi-dimensional failure time model for petascale supercomputers.

A Method to Analyze and Optimize Parallel Programs on CPU/MIC Heterogeneous Architecture

Cooperating with the Intel® Many Integrated Core Architecture which was announced in 2010 as a massively parallel coprocessor, heterogeneous node has been broadly applied in petascale supercomputers, such as TianHe-II. The performance analysis for massive parallel applications under the heterogeneous architecture is playing an important role for next generation exascale supercomputers. In this paper, we proposed a method to analyze and optimize parallel programs running on CPU/MIC heterogeneous computing node with offload programming mode. To monitor runtime behaviors in offload process, we used TAU to instrument the offload code region to collect performance events. Then we compared the performance of different programming modes with NAS Parallel Benchmarks. The results indicated that asynchronous offload mode performs better than synchronous offload mode.

Massive Computation for Understanding Core-Collapse Supernova Explosions

How do massive stars explode? Progress toward the answer is driven by increases in compute power. Petascale supercomputers are enabling detailed three-dimensional simulations of core-collapse supernovae. These are elucidating the role of fluid instabilities, turbulence, and magnetic field amplification in supernova engines.

Parallelization of 3-D Global FDTD Earth-Ionosphere Waveguide Models at Resolutions on the Order of ∼1 km and Higher

Three-dimensional finite-difference time-domain (FDTD) models of the global Earth-ionosphere waveguide were first introduced in the early 2000s. These models have been applied to a wide variety of applications at ultralow and extremely low frequencies, including hypothesized earthquake precursors, Schumann resonances, and space weather hazards to electric power grids. The deployment of petascale supercomputers and future plans for exascale supercomputers introduce for the first time the possibility of generating ultrahigh-resolution FDTD models of the Earth-ionosphere waveguide at 1 km and higher globally. When including a magnetized ionospheric plasma algorithm, these models may simulate very low frequencies for the first time on a global scale and/or extend to altitudes well above 100 km as needed. We describe the parallelization strategy of efficient latitude–longitude FDTD Earth-ionosphere models and describe some example performances on the Blue Waters supercomputer, which has over 360 000 computational cores.

Massively Parallel Algorithms for the Lattice Boltzmann Method on NonUniform Grids

The lattice Boltzmann method exhibits excellent scalability on current supercomputing systems and has thus increasingly become an alternative method for large-scale non-stationary flow simulations, reaching up to a trillion grid nodes. Additionally, grid refinement can lead to substantial savings in memory and compute time. These saving, however, come at the cost of much more complex data structures and algorithms. In particular, the interface between subdomains with different grid sizes must receive special treatment. In this article, we present parallel algorithms, distributed data structures, and communication routines that are implemented in the software framework waLBerla in order to support large-scale, massively parallel lattice Boltzmann-based simulations on non-uniform grids. Additionally, we evaluate the performance of our approach on two current petascale supercomputers. On an IBM Blue Gene/Q system, the largest weak scaling benchmarks with refined grids are executed with almost two million threads, demonstrating not only near-perfect scalability but also an absolute performance of close to a trillion lattice Boltzmann cell updates per second. On an Intel-based system, the strong scaling of a simulation with refined grids and a total of more than 8.5 million cells is demonstrated to reach a performance of less than one millisecond per time step. This enables simulations with complex, non-uniform grids and four million time steps per hour compute time.

Ultrascale simulations of non-smooth granular dynamics

This article presents new algorithms for massively parallel granular dynamics simulations on distributed memory architectures using a domain partitioning approach. Collisions are modelled with hard contacts in order to hide their micro-dynamics and thus to extend the time and length scales that can be simulated. The global multi-contact problem is solved using a non-linear block Gauss-Seidel method that is conforming to the subdomain structure. The parallel algorithms employ a sophisticated protocol between processors that delegate algorithmic tasks such as contact treatment and position integration uniquely and robustly to the processors. Communication overhead is minimized through aggressive message aggregation, leading to excellent strong and weak scaling. The robustness and scalability is assessed on three clusters including two peta-scale supercomputers with up to 458,752 processor cores. The simulations can reach unprecedented resolution of up to ten billion ( $$10^{10}$$ ) non-spherical particles and contacts.

Mapping to Irregular Torus Topologies and Other Techniques for Petascale Biomolecular Simulation.

Currently deployed petascale supercomputers typically use toroidal network topologies in three or more dimensions. While these networks perform well for topology-agnostic codes on a few thousand nodes, leadership machines with 20,000 nodes require topology awareness to avoid network contention for communication-intensive codes. Topology adaptation is complicated by irregular node allocation shapes and holes due to dedicated input/output nodes or hardware failure. In the context of the popular molecular dynamics program NAMD, we present methods for mapping a periodic 3-D grid of fixed-size spatial decomposition domains to 3-D Cray Gemini and 5-D IBM Blue Gene/Q toroidal networks to enable hundred-million atom full machine simulations, and to similarly partition node allocations into compact domains for smaller simulations using multiple-copy algorithms. Additional enabling techniques are discussed and performance is reported for NCSA Blue Waters, ORNL Titan, ANL Mira, TACC Stampede, and NERSC Edison.

Spiking network simulation code for petascale computers.

Brain-scale networks exhibit a breathtaking heterogeneity in the dynamical properties and parameters of their constituents. At cellular resolution, the entities of theory are neurons and synapses and over the past decade researchers have learned to manage the heterogeneity of neurons and synapses with efficient data structures. Already early parallel simulation codes stored synapses in a distributed fashion such that a synapse solely consumes memory on the compute node harboring the target neuron. As petaflop computers with some 100,000 nodes become increasingly available for neuroscience, new challenges arise for neuronal network simulation software: Each neuron contacts on the order of 10,000 other neurons and thus has targets only on a fraction of all compute nodes; furthermore, for any given source neuron, at most a single synapse is typically created on any compute node. From the viewpoint of an individual compute node, the heterogeneity in the synaptic target lists thus collapses along two dimensions: the dimension of the types of synapses and the dimension of the number of synapses of a given type. Here we present a data structure taking advantage of this double collapse using metaprogramming techniques. After introducing the relevant scaling scenario for brain-scale simulations, we quantitatively discuss the performance on two supercomputers. We show that the novel architecture scales to the largest petascale supercomputers available today.

A More Scalable and Efficient Parallelization of the Adaptive Integral Method—Part II: BIOEM Application

The parallelization of the adaptive integral method proposed in Part I is used to solve 3-D scattering problems pertinent to bioelectromagnetic (BIOEM) analysis. Detailed numerical results are presented to quantify the computational complexity and parallel efficiency of the method on a petascale supercomputing cluster. Boundaries of acceptable parallelization regions of the method are identified in the N - P plane under realistic resource and efficiency constraints, where N and P denote the number of unknowns and processes, respectively. These regions are shown to be larger than those of an alternative parallelization method for benchmark BIOEM problems. The proposed method is also used to compute the power absorbed by an anatomical human body model to demonstrate its potential for solving complex BIOEM problems. The computations are performed for increasingly higher resolutions of the model and the power absorbed by the entire model and specific tissues in it are compared to safety standards. The highest resolution body model leads to an extreme problem with about 1.2 billion unknowns; the absolute parallel efficiency of the power-absorption simulation with this model is shown to be approximately 78% for matrix fill operations, 66% for memory requirement, and 30% for iterative solution on 8192 processes.

Application-specific I/O Optimizations on Petascale Supercomputers

Data-intensive science frontiers and challenges are emerg ing as computer technology has evolved substantially. Large-scale simu lations demand significant I/O workload, and as a result the I/O performance often becomes a bottleneck preventing high performance in scientific applications. In this paper we introduce a variety of I/O optimization techniques developed and imple mented when scaling a seismic application to petascale. These techniques include file system striping, data aggreg ation, reader/writer limit ing and less interleaving of data, collective MPI-IO, and data staging. The optimizations result in nearly perfect scalability of the target applicat ion on some of the most advanced petascale systems. The techniques introduced in this paper are applicab le to other scientific applications facing similar petascale I/O challenges.

Exploiting multi-scale parallelism for large scale numerical modelling of laser wakefield accelerators

A new generation of laser wakefield accelerators (LWFA), supported by the extreme accelerating fields generated in the interaction of PW-Class lasers and underdense targets, promises the production of high quality electron beams in short distances for multiple applications. Achieving this goal will rely heavily on numerical modelling to further understand the underlying physics and identify optimal regimes, but large scale modelling of these scenarios is computationally heavy and requires the efficient use of state-of-the-art petascale supercomputing systems. We discuss the main difficulties involved in running these simulations and the new developments implemented in the OSIRIS framework to address these issues, ranging from multi-dimensional dynamic load balancing and hybrid distributed/shared memory parallelism to the vectorization of the PIC algorithm. We present the results of the OASCR Joule Metric program on the issue of large scale modelling of LWFA, demonstrating speedups of over 1 order of magnitude on the same hardware. Finally, scalability to over ∼106 cores and sustained performance over ∼2 P Flops is demonstrated, opening the way for large scale modelling of LWFA scenarios.

MEGADOCK 3.0: a high-performance protein-protein interaction prediction software using hybrid parallel computing for petascale supercomputing environments

BackgroundProtein-protein interaction (PPI) plays a core role in cellular functions. Massively parallel supercomputing systems have been actively developed over the past few years, which enable large-scale biological problems to be solved, such as PPI network prediction based on tertiary structures.ResultsWe have developed a high throughput and ultra-fast PPI prediction system based on rigid docking, “MEGADOCK”, by employing a hybrid parallelization (MPI/OpenMP) technique assuming usages on massively parallel supercomputing systems. MEGADOCK displays significantly faster processing speed in the rigid-body docking process that leads to full utilization of protein tertiary structural data for large-scale and network-level problems in systems biology. Moreover, the system was scalable as shown by measurements carried out on two supercomputing environments. We then conducted prediction of biological PPI networks using the post-docking analysis.ConclusionsWe present a new protein-protein docking engine aimed at exhaustive docking of mega-order numbers of protein pairs. The system was shown to be scalable by running on thousands of nodes. The software package is available at: http://www.bi.cs.titech.ac.jp/megadock/k/.

Flexible composition and execution of high performance, high fidelity multiscale biomedical simulations

Multiscale simulations are essential in the biomedical domain to accurately model human physiology. We present a modular approach for designing, constructing and executing multiscale simulations on a wide range of resources, from laptops to petascale supercomputers, including combinations of these. Our work features two multiscale applications, in-stent restenosis and cerebrovascular bloodflow, which combine multiple existing single-scale applications to create a multiscale simulation. These applications can be efficiently coupled, deployed and executed on computers up to the largest (peta) scale, incurring a coupling overhead of 1-10% of the total execution time.

Applying High-performance Computing to Petascale Explosive Simulations

Hazardous scenarios involving explosives are difficult to experimentally study and simulation is often the only viable approach to study highly reactive phenomena. Explosive simulations are computationally expensive, requiring supercomputing resources for continued scientific discovery in the field. Here an idealized mesoscale simulation of explosive grains under mechanical insult by a high-speed projectile with reaction represented by a novel kinetic model is designed to test the scalability of the Uintah software on petascale supercomputers. Good scalability is found up to 49K processors. Timing breakdown of compu- tational tasks are determined with relocation of Lagrangian particles and interpolation of those particles to the grid identified as the most expensive operation and ideal for optimization. Potential optimization strategies are identified. Realistic model simulations rather than toy model simulations are found to better represent scalability of a science code on a supercomputer. Estimations for total supercomputer hours necessary to complete the kinetic model validation study are reported.