Petascale Simulations Research Articles

Advanced petascale simulations of the scaling up of mixing limited flow processes for materials synthesis

The paper focuses on the use of petascale simulations to address the open-ended problem of scaling-up continuous flow processes in microfluidics. Some of these processes are mixing limited and so require deep analysis of the scale up effects onto the mixing performance. The scaling up strategy considered is the so-called sizing-up approach based on the conservation of the turbulent dissipation rate in the reactor combined with petascale simulations. High fidelity Direct Numerical Simulations are performed from micro- to milli-scale reactor (up to 11 billion of mesh nodes on 131,072 processors) while maintaining the same accuracy of the physical phenomena description (smaller dissipative scales). It leads to confident qualitative estimations of the mixing performance for the different sizing-up scenarios. We show that it is possible to increase up to two orders of magnitude the production rate of the continuous reactor while maintaining an excellent mixing quality, which is very favorable for the material synthesis process.

Toward Exascale: Overview of Large Eddy Simulations and Direct Numerical Simulations of Nuclear Reactor Flows with the Spectral Element Method in Nek5000

At the beginning of the last decade, Petascale supercomputers (i.e., computers capable of more than 1 petaFLOP) emerged. Now, at the dawn of exascale supercomputing, we provide a review of recent landmark simulations of portions of reactor components with turbulence-resolving techniques that this computational power has made possible. In fact, these simulations have provided invaluable insight into flow dynamics, which is difficult or often impossible to obtain with experiments alone. We focus on simulations performed with the spectral element method, as this method has emerged as a powerful tool to deliver massively parallel calculations at high fidelity by using large eddy simulation or direct numerical simulation. We also limit this paper to constant-property incompressible flow of a Newtonian fluid in the absence of other body or external forces, although the method is by no means limited to this class of flows. We briefly review the fundamentals of the method and the reasons it is compelling for the simulation of nuclear engineering flows. We review in detail a series of Petascale simulations, including the simulations of helical coil steam generators, fuel assemblies, and pebble beds. Even with Petascale computing, however, limitations for nuclear modeling and simulation tools remain. In particular, the size and scope of turbulence-resolving simulations are still limited by computing power and resolution requirements, which scale with the Reynolds number. In the final part of this paper, we discuss the future of the field, including recent advancements in emerging architectures such as GPU-based supercomputers, which are expected to power the next generation of high-performance computers.

Petascale simulations of compressible flows with interfaces

We demonstrate a high throughput software for the efficient simulation of compressible multicomponent flow on high performance computing platforms. The discrete problem is represented on structured three-dimensional grids with non-uniform resolution. Discontinuous flow features are captured using a diffuse interface method. A distinguishing characteristic of the method is the proper treatment of the interface zone as a mixing region of liquid and gas. The governing equations are discretized by a Godunov-type finite volume method with explicit time stepping using a low-storage Runge-Kutta scheme. The presented flow solver Cubism-MPCF is based on our Cubism library which enables a highly optimized framework for the efficient treatment of stencil based problems on multicore architectures. The framework is general and not limited to applications in fluid dynamics. We validate our solver by classical benchmark examples. Furthermore, we examine a highly-resolved shock-induced bubble collapse and a cloud of O(103) collapsing bubbles, which demonstrate the high potential of the proposed framework and solver.

A dual communicator and dual grid-resolution algorithm for petascale simulations of turbulent mixing at high Schmidt number

A new dual-communicator algorithm with very favorable performance characteristics has been developed for direct numerical simulation (DNS) of turbulent mixing of a passive scalar governed by an advection–diffusion equation. We focus on the regime of high Schmidt number (Sc), where because of low molecular diffusivity the grid-resolution requirements for the scalar field are stricter than those for the velocity field by a factor Sc. Computational throughput is improved by simulating the velocity field on a coarse grid of Nv3 points with a Fourier pseudo-spectral (FPS) method, while the passive scalar is simulated on a fine grid of Nθ3 points with a combined compact finite difference (CCD) scheme which computes first and second derivatives at eighth-order accuracy. A static three-dimensional domain decomposition and a parallel solution algorithm for the CCD scheme are used to avoid the heavy communication cost of memory transposes. A kernel is used to evaluate several approaches to optimize the performance of the CCD routines, which account for 60% of the overall simulation cost. On the petascale supercomputer Blue Waters at the University of Illinois, Urbana–Champaign, scalability is improved substantially with a hybrid MPI-OpenMP approach in which a dedicated thread per NUMA domain overlaps communication calls with computational tasks performed by a separate team of threads spawned using OpenMP nested parallelism. At a target production problem size of 81923 (0.5 trillion) grid points on 262,144 cores, CCD timings are reduced by 34% compared to a pure-MPI implementation. Timings for 163843 (4 trillion) grid points on 524,288 cores encouragingly maintain scalability greater than 90%, although the wall clock time is too high for production runs at this size. Performance monitoring with CrayPat for problem sizes up to 40963 shows that the CCD routines can achieve nearly 6% of the peak flop rate. The new DNS code is built upon two existing FPS and CCD codes. With the grid ratio Nθ∕Nv=8, the disparity in the computational requirements for the velocity and scalar problems is addressed by splitting the global communicator MPI_COMM_WORLD into disjoint communicators for the velocity and scalar fields, respectively. Inter-communicator transfer of the velocity field from the velocity communicator to the scalar communicator is handled with discrete send and non-blocking receive calls, which are overlapped with other operations on the scalar communicator. For production simulations at Nθ=8192 and Nv=1024 on 262,144 cores for the scalar field, the DNS code achieves 94% strong scaling relative to 65,536 cores and 92% weak scaling relative to Nθ=1024 and Nv=128 on 512 cores.

Optimising the Termofluids CFD code for petascale simulations

ABSTRACTThis paper presents some recent efforts carried out on the expansion of the scalability of TermoFluids multi-physics Computational Fluid Dynamics (CFD) code, aiming to achieve petascale capacity for a single simulation. We describe different aspects that we have improved in our code in order to efficiently run it on 131,072 CPU-cores. This work has been developed using the BlueGene/Q Mira supercomputer of the Argonne Leadership Computing Facility, where we have obtained feedback at the targeted scale. In summary, this is a practical paper showing our experience at reaching the petascale paradigm for a single simulation with TermoFluids.

Petascale Simulations of the Morphology and the Molecular Interface of Bulk Heterojunctions.

Understanding how additives interact and segregate within bulk heterojunction (BHJ) thin films is critical for exercising control over structure at multiple length scales and delivering improvements in photovoltaic performance. The morphological evolution of poly(3-hexylthiophene) (P3HT) and phenyl-C61-butyric acid methyl ester (PCBM) blends that are commensurate with the size of a BHJ thin film is examined using petascale coarse-grained molecular dynamics simulations. Comparisons between two-component and three-component systems containing short P3HT chains as additives undergoing thermal annealing demonstrate that the short chains alter the morphology in apparently useful ways: they efficiently migrate to the P3HT/PCBM interface, increasing the P3HT domain size and interfacial area. Simulation results agree with depth profiles determined from neutron reflectometry measurements that reveal PCBM enrichment near substrate and air interfaces but a decrease in that PCBM enrichment when a small amount of short P3HT chains are integrated into the BHJ blend. Atomistic simulations of the P3HT/PCBM blend interfaces show a nonmonotonic dependence of the interfacial thickness as a function of number of repeat units in the oligomeric P3HT additive, and the thiophene rings orient parallel to the interfacial plane as they approach the PCBM domain. Using the nanoscale geometries of the P3HT oligomers, LUMO and HOMO energy levels calculated by density functional theory are found to be invariant across the donor/acceptor interface. These connections between additives, processing, and morphology at all length scales are generally useful for efforts to improve device performance.

A high order finite difference solver for massively parallel simulations of stably stratified turbulent channel flows

Investigation of stably stratified turbulent flows in the atmosphere is challenging owing to its nonstationarity and heterogeneity, especially in very stable conditions. In recent years, numerical simulation has become a powerful tool to complement the understanding of fundamental characteristics of strongly stratified flows. Owing to its highly accurate representation of small-scale turbulence structures, direct numerical simulation (DNS) is shown to be the most effective numerical approach for strongly stratified flows. However, due to the limitation of the traditional parallel strategy, existing DNS studies on stratified channel flows are limited to relatively low Reynolds numbers. In order to make the DNS studies more relevant to the atmospheric boundary layer flows, in this paper, a high-order finite-difference DNS solver is developed for simulating stably stratified turbulent channel flows at relatively high Reynolds numbers. Utilizing the advanced two dimensional domain decomposition technique, the DNS solver is shown to be competent for petascale simulations with excellent scalability tested up to 32,768 CPU cores. Using this newly developed solver, two stably stratified open-channel simulations are conducted with the highest Reynolds number being approximately 105. The fundamental characteristics of strongly stratified turbulent flows, e.g., spatio-temporal variation of global intermittency, are compared with those previously reported at relatively low Reynolds numbers, and their similarities and discrepancies are also discussed. In addition, the DNS-based Monin–Obukhov similarity relationships are compared with those proposed based on observational data to evaluate their relevance to atmospheric flows. The source code and the simulation results are available at http://sites.google.com/site/scdnscode/.

Numerical strategies towards peta-scale simulations of nanoelectronics devices

We address two challenges with the development of next-generation nanotransistors, (i) the capability of modeling realistically extended structures on an atomistic basis and (ii) predictive simulations that are faster and cheaper than experiments. We have developed a multi-dimensional, quantum transport solver, OMEN, towards these goals. To approach the peta-scale, the calculation of the open boundary conditions connecting the simulation domain to its environment is interleaved with the computation of the device wave functions and the work load of each task is predicted prior to any calculation, resulting in a dynamic core allocation. OMEN uses up to 147,456 cores on Jaguar with four levels of MPI parallelization and reaches a sustained performance of 504 TFlop/s, running at 37% of the machine peak performance. We investigate 3D nanowire transistors with diameters up to 10 nm, reproduce experimental data of high electron mobility 2D transistors, and expect increased capabilities by using over 300,000 cores in the future.

Shock-turbulence interaction: What we know and what we can learn from peta-scale simulations

Many applications in engineering and physical sciences involve turbulent flows interacting with shock waves. High-speed flows around aerodynamic bodies and through propulsion systems for high-speed flight abound with interactions of shear driven turbulence with complex shock waves. Supernova explosions and implosion of a cryogenic fuel pellet for inertial confinement fusion also involve the interaction of shockwaves with turbulence and strong density variations. Numerical simulations of such physical phenomena impose conflicting demands on the numerical algorithms. Capturing broadband spatial and temporal variations in a turbulent flow suggests the use of high-bandwidth schemes with minimal dissipation and dispersion, while capturing the flow discontinuity at a shock wave requires numerical dissipation.Results from three promising shock-capturing schemes a) high order WENO, b) nonlinear artificial diffusivity with compact finite differences, and c) a hybrid approach combining high-order central differencing with WENO near the shocks are compared using the Taylor-Green problem and compressible isotropic turbulence with eddy-shocklets. The performance of each scheme is characterized in terms of an effective bandwidth. The comparison highlights the damaging effect of numerical dissipation when the WENO scheme is applied everywhere. The hybrid approach is found to be best suited for studying shock-turbulence interactions.Results from previous DNS and LES studies of the canonical shock-turbulence interaction problem, i.e. the interaction of isotropic turbulence with a (nominally) normal shock and comparison with available theory and experimental data are recalled. The principal physical effects include the amplification of turbulent kinetic energy across the shock and its anisotropy, change in turbulence length scales across the shock, departure from the common assumption of strong Reynolds analogy (used in modeling turbulence in high-speed flows), and the distortion of the shock due to its interaction with the incident turbulence. In this context new results from our SciDAC sponsored project are shown and open issues are mentioned. New DNS results achieve a significantly larger turbulence Reynolds number and allow an exploration of the nonlinear effects. While the linear interaction theory of Ribner provides useful estimates of the amplification of vorticity fluctuations across the shock, it misses the strong nonlinear dynamics of the energized and highly anisotropic vorticity downstream of the shock. It is found that previous DNS studies also underestimated this effect. The simulations show that turbulent self-stretching and tilting mechanisms bring about a relatively rapid return to isotropy in the turbulent vorticity field. The turbulent velocity field, however, does not show any appreciable tendency towards isotropy. It is further observed that when the turbulence interacting with the shock is sufficiently energetic the instantaneous shock structure is significantly modified; local regions of significant over-compression are found as well as regions where the mean shock compression is nearly isentropic. Estimates for the computational resources necessary for studying this fundamental shock-turbulence interaction problem at higher Reynolds number on peta-scale computing systems are given.

Combinatorial algorithms for computational science and engineering

We discuss recent activities in the SciDAC Applied Mathematics Institute for Combinatorial Scientific Computing and Petascale Simulations (CSCAPES).

Scalable solver software

AbstractTowards Optimal Petascale Simulations (TOPS) is a scalable solver software project based on domain decomposed parallelization to research, implement, and support in collaborations with users an open‐source package for large‐scale discretized PDE problems. Optimal complexity methods, such as multigrid/multilevel preconditioners, keep the time spent in dominant algebraic kernels close to linear in discrete problem size as the applications scale on massively parallel computers. Krylov accelerators and Jacobian‐free variants of Newton's method, as appropriate, are wrapped around the multilevel methods to deliver robustness in multirate, multiscale coupled systems, which are solved either implicitly or in more traditional forms of operator splitting. The TOPS software framework is being extended beyond direct computational simulation to computational optimization, including design, control, and inverse problems. We outline and illustrate the philosophy of TOPS. (© 2008 WILEY‐VCH Verlag GmbH & Co. KGaA, Weinheim)

Enabling high performance computational science through combinatorial algorithms

The Combinatorial Scientific Computing and Petascale Simulations (CSCAPES) Institute is developing algorithms and software for combinatorial problems that play an enabling role in scientific and engineering computations. Discrete algorithms will be increasingly critical for achieving high performance for irregular problems on petascale architectures. This paper describes recent contributions by researchers at the CSCAPES Institute in the areas of load balancing, parallel graph coloring, performance improvement, and parallel automatic differentiation.

Interoperable mesh and geometry tools for advanced petascale simulations

SciDAC applications have a demonstrated need for advanced software tools to manage the complexities associated with sophisticated geometry, mesh, and field manipulation tasks, particularly as computer architectures move toward the petascale. The Center for Interoperable Technologies for Advanced Petascale Simulations (ITAPS) will deliver interoperable and interchangeable mesh, geometry, and field manipulation services that are of direct use to SciDAC applications. The premise of our technology development goal is to provide such services as libraries that can be used with minimal intrusion into application codes. To develop these technologies, we focus on defining a common data model and data-structure neutral interfaces that unify a number of different services such as mesh generation and improvement, front tracking, adaptive mesh refinement, shape optimization, and solution transfer operations. We highlight the use of several ITAPS services in SciDAC applications.