Adaptive Mesh Refinement Capability Research Articles

Towards exascale for wind energy simulations

We examine large-eddy-simulation modeling approaches and computational performance of two open-source computational fluid dynamics codes for the simulation of atmospheric boundary layer flows that are of direct relevance to wind energy production. The first code, NekRS, is a high-order, unstructured-grid, spectral element code. The second code, AMR-Wind, is a second-order, block-structured, finite-volume code with adaptive mesh refinement capabilities. The objective of this study is to co-develop these codes in order to improve model fidelity and performance for each. These features will be critical for running ABL-based applications such as wind farm analysis on advanced computing architectures. To this end, we investigate the performance of NekRS and AMR-Wind on the Oak Ridge Leadership Facility supercomputers Summit, using 4 to 800 nodes (24 to 4,800 NVIDIA V100 GPUs), and Crusher, the testbed for the Frontier exascale system, using 18 to 384 Graphics Compute Dies on AMD MI250X GPUs. We compare strong- and weak-scaling capabilities, linear solver performance, and time to solution. We also identify leading inhibitors to parallel scaling.

Computational Techniques to Generate Space Launch System Aerodynamic Databases

This document describes the reasoning and trade studies used to evaluate tools for constructing the aerodynamic lineload databases for the liftoff and transition phases of flight for the Space Launch System. Three computational fluid dynamics codes (USM3D, FUN3D, and Kestrel) were investigated with various turbulence models as well as detached eddy simulation variants for the launch vehicle in free air and in proximity to the tower. Decisions were made mostly based on results from brief developmental studies performed in response to specific, unforeseen challenges that were encountered in the analysis of a given configuration. The need to develop databases in a timely manner, as well as to accurately capture the expected leeward-wake flowfield characteristics, led to the selection of the Kestrel flow solver with its delayed detached eddy simulation method, the Spalart–Allmaras turbulence model, and the adaptive mesh refinement capability in the off-body Cartesian grid region.

Adaptive mesh refinement in locally conservative level set methods for multiphase fluid displacements in porous media

Multiphase flow in porous media often occurs with the formation and coalescence of fluid ganglia. Accurate predictions of such mechanisms in complex pore geometries require simulation models with local mass conservation and with the option to improve resolution in areas of interest. In this work, we incorporate patch-based, structured adaptive mesh refinement capabilities into a method for local volume conservation that describes the behaviour of disconnected fluid ganglia during level set simulations of capillary-controlled displacement in porous media. We validate the model against analytical solutions for three-phase fluid configurations in idealized pores containing gas, oil, and water, by modelling the intermediate-wet oil layers as separate domains with their volumes preserved. Both the pressures and volumes of disconnected ganglia converge to analytical values with increased refinement levels of the adaptive mesh. Favourable results from strong and weak scaling tests emphasize that the number of patches per processor and the total number of patches are important parameters for efficient parallel simulations with adaptive mesh refinement. Simulations of two-phase imbibition and three-phase gas invasion on segmented 3D images of water-wet sandstone show that adaptive mesh refinement has the highest impact on three-phase displacements, especially concerning the behaviour of the conserved, intermediate-wet phase.

Vortexlet formation in Schardin's problem

The present study focuses on the shock diffraction problem over a triangle wedge for Mach numbers of M=1.3, 1.5, 1.7, and 2.0 by using a two-dimensional, high-order, in-house Euler solver. The solver is based on a family of advection upstream splitting method in combination with a central essentially non-oscillatory scheme and benefits a block-based adaptive mesh refinement algorithm to resolve the regions that contain discontinuities. High accuracies in time and space, and adaptive mesh refinement capabilities of the solver allow us to investigate vortexlet formation mechanism in detail. Our results reveal that there are two different types of vortexlet formation mechanisms. While the first type of formation is observed at all Mach numbers considered here, the second type arises when the Mach number is greater than 1.3. This difference results from their driving mechanisms, which are the upward moving accelerated shock and embedded shock in the primary vortex. In addition to their driving mechanisms, two types are also different in terms of their locations.

An efficient IMEX-DG solver for the compressible Navier-Stokes equations for non-ideal gases

We propose an efficient, accurate and robust IMEX solver for the compressible Navier-Stokes equations describing non-ideal gases with general cubic equation of state and Stiffened-Gas EOS. The method is based on an h-adaptive Discontinuous Galerkin spatial discretization and on an Additive Runge Kutta IMEX method for time discretization. It is specifically tailored for low Mach number applications and allows to simulate low Mach regimes at a significantly reduced computational cost, while maintaining full second order accuracy also for higher Mach number regimes. The method has been implemented in the framework of the deal.II numerical library, whose adaptive mesh refinement capabilities are employed to enhance efficiency. Refinement indicators appropriate for real gas phenomena have been introduced. A number of numerical experiments on classical benchmarks for compressible flows and their extension to real gases demonstrate the properties of the proposed method.

Gravity versus Magnetic Fields in Forming Molecular Clouds

Abstract Magnetic fields are dynamically important in the diffuse interstellar medium. Understanding how gravitationally bound, star-forming clouds form requires modeling of the fields in a self-consistent, supernova-driven, turbulent, magnetized, stratified disk. We employ the FLASH magnetohydrodynamics code to follow the formation and early evolution of clouds with final masses of 3–8 × 103 M ⊙ within such a simulation. We use the code’s adaptive mesh refinement capabilities to concentrate numerical resolution in zoom-in regions covering single clouds, allowing us to investigate the detailed dynamics and field structure of individual self-gravitating clouds in a consistent background medium. Our goal is to test the hypothesis that dense clouds are dynamically evolving objects far from magnetohydrostatic equilibrium. We find that the cloud envelopes are magnetically supported with field lines parallel to density gradients and flow velocity, as indicated by the histogram of relative orientations and other statistical measures. In contrast, the dense cores of the clouds are gravitationally dominated, with gravitational energy exceeding internal, kinetic, or magnetic energy and accelerations due to gravity exceeding those due to magnetic or thermal pressure gradients. In these regions, field directions vary strongly, with a slight preference toward being perpendicular to density gradients, as shown by three-dimensional histograms of relative orientation.

A stable space-time FE method for the shallow water equations

We consider the finite element (FE) approximation of the shallow water equations (SWE) by considering discretizations in which both space and time are established using an unconditionally stable FE method. Particularly, we consider the automatic variationally stable FE (AVS-FE) method, a type of discontinuous Petrov-Galerkin (DPG) method. The philosophy of the DPG method allows us to break the test space and achieve unconditionally stable FE approximations as well as accurate a posteriori error estimators upon solution of a saddle point system of equations. The resulting error indicators allow us to employ mesh adaptive strategies and perform space-time mesh refinements, i.e., local time stepping. We derive a priori error estimates for the AVS-FE method and linearized SWE and perform numerical verifications to confirm corresponding asymptotic convergence behavior. In an effort to keep the computational cost low, we consider an alternative space-time approach in which the space-time domain is partitioned into finite sized space-time slices. Hence, we can perform adaptivity on each individual slice to preset error tolerances as needed for a particular application. Numerical verifications comparing the two alternatives indicate the space-time slices are superior for simulations over long times, whereas the solutions are indistinguishable for short times. Multiple numerical verifications show the adaptive mesh refinement capabilities of the AVS-FE method, as well the application of the method to commonly applied benchmarks for the SWE.

GR-Athena++: Puncture Evolutions on Vertex-centered Oct-tree Adaptive Mesh Refinement

Abstract Numerical relativity is central to the investigation of astrophysical sources in the dynamical and strong-field gravity regime, such as binary black hole and neutron star coalescences. Current challenges set by gravitational-wave and multimessenger astronomy call for highly performant and scalable codes on modern massively parallel architectures. We present GR-Athena++, a general-relativistic, high-order, vertex-centered solver that extends the oct-tree, adaptive mesh refinement capabilities of the astrophysical (radiation) magnetohydrodynamics code Athena++. To simulate dynamical spacetimes, GR-Athena++ uses the Z4c evolution scheme of numerical relativity coupled to the moving puncture gauge. We demonstrate stable and accurate binary black hole merger evolutions via extensive convergence testing, cross-code validation, and verification against state-of-the-art effective-one-body waveforms. GR-Athena++ leverages the task-based parallelism paradigm of Athena++ to achieve excellent scalability. We measure strong-scaling efficiencies above 95% for up to ∼1.2 × 104 CPUs and excellent weak scaling is shown up to ∼105 CPUs in a production binary black hole setup with adaptive mesh refinement. GR-Athena++ thus allows for the robust simulation of compact binary coalescences and offers a viable path toward numerical relativity at exascale.

Efficient Simulations of Propagating Flames and Fire Suppression Optimization Using Adaptive Mesh Refinement

Fires are complex multi-physics problems that span wide spatial scale ranges. Capturing this complexity in computationally affordable numerical simulations for process studies and “outer-loop” techniques (e.g., optimization and uncertainty quantification) is a fundamental challenge in reacting flow research. Further complications arise for propagating fires where a priori knowledge of the fire spread rate and direction is typically not available. In such cases, static mesh refinement at all possible fire locations is a computationally inefficient approach to bridging the wide range of spatial scales relevant to fire behavior. In the present study, we address this challenge by incorporating adaptive mesh refinement (AMR) in fireFoam, an OpenFOAM solver for simulations of complex fire phenomena involving pyrolyzing solid surfaces. The AMR functionality in the extended solver, called fireDyMFoam, is load balanced, models gas, solid, and liquid phases, and allows us to dynamically track regions of interest, thus avoiding inefficient over-resolution of areas far from a propagating flame. We demonstrate the AMR capability and computational efficiency for fire spread on vertical panels, showing that the AMR solver reproduces results obtained using much larger statically refined meshes, but at a substantially reduced computational cost. We then leverage AMR in an optimization framework for fire suppression based on the open-source Dakota toolkit, which is made more computationally tractable through the use of fireDyMFoam, minimizing a cost function that balances water use and solid-phase mass loss. The extension of fireFoam developed here thus enables the use of higher fidelity simulations in optimization problems for the suppression of fire spread in both built and natural environments.

Multicomponent fluid model for two-temperature plasmas derived from kinetic theory : application to magnetic reconnection

This contribution deals with the fluid modeling of multicomponent magnetized plasmas in thermo-chemical non-equilibrium from the partially- to fully-ionized collisional regimes, aiming at the predictive simulation of magnetic reconnection in Sun chromosphere conditions. Such fluid models are required for large-scale simulations by relying on high performance computing. The fluid model is derived from a kinetic theory approach, yielding a rigorous description of the dissipative and non-equilibrium effects and a well-identified mathematical structure. We start from a general system of equations that is obtained by means of a multiscale Chapman-Enskog method, based on a non-dimensional analysis accounting for the mass disparity between the electrons and heavy particles, including the influence of the electromagnetic field and transport properties. The latter are computed by using a spectral Galerkin method based on a converged Laguerre-Sonine polynomial approximation. Then, in the limit of small Debye length with respect to the characteristic scale in the Sun chromosphere, we derive a two-temperature single-momentum multicomponent diffusion model coupled to Maxwell’s equations, which is able to describe fully- and partially-ionized plasmas, beyond the multi-fluid model of Braginskii, valid for the whole range of the Sun chromosphere conditions. The second contribution is the development and verification of an accurate and robust numerical strategy that is based on CanoP, a massively parallel code with adaptive mesh refinement capability, which is able to cope with the full spectrum of scales of the magnetic reconnection process, without additional constraint on the time steps compared to single-fluid Magnetohydrodynamics (MHD) models. The final contribution is a study of the physics of magnetic reconnection in collaboration with the heliophysics team of NASA Ames Research Center. We show that the model and methods allow us to retrieve the results of usual single-fluid MHD models in the highly collisional case at equilibrium, while achieving a more detailed physics description relevant to such applications in the weakly collisional case, where non-equilibrium effects become important.

Binarized-octree generation for Cartesian adaptive mesh refinement around immersed geometries

We revisit the generation of balanced octrees for adaptive mesh refinement (AMR) of Cartesian domains with immersed complex geometries. In a recent short note (Hasbestan and Senocak, 2017) [42], we showed that the data locality of the Z-order curve in a hashed linear-octree generation method may not be perfect because of potential collisions in the hash table. Building on that observation, we propose a binarized-octree generation method that complies with the Z-order curve exactly. Similar to a hashed linear-octree generation method, we use Morton encoding to index the nodes of an octree, but use a red-black tree in place of the hash table. Red-black tree is a special kind of a binary tree, which we use for insertion and deletion of elements during mesh adaptation. By strictly working with the bitwise representation of an octree, we remove computer hardware limitations on the depth of adaptation on a single processor. Additionally, we introduce a geometry encoding technique for rapidly tagging a solid geometry for mesh refinement. Our results for several geometries with different levels of adaptations show that the binarized-octree generation method outperforms the linear-octree generation method in terms of runtime performance at the expense of only a slight increase in memory usage. The current AMR capability, rebl-AMR, is available as open-source software.

Simulations of Ion Thruster Plumes in Ground Facilities Using Adaptive Mesh Refinement

Ion thruster plumes are simulated under a recently developed computer framework inside a vacuum chamber at various background pressures for single- and triple-thruster configurations. Momentum and charge–exchange collisions occurring between neutral and ion species, as well as the induced electric field due to ions, are performed using multiple adaptive mesh refinement meshes to study the spatial variance of neutrals and ions while analyzing the effect of vacuum chamber conditions. Comparisons between vacuum chamber and space simulations show that the spatial distribution of the neutral densities are different in the far downstream region of the ion thruster. The change in the background density is also found to affect the spatial distributions of the charge–exchange ions in the domain. The triple-thruster cases show that the background density levels are approximately proportional to the number of thrusters being operated. The adaptive mesh refinement capability is shown to be able to capture the confinement effect of the flow in the vacuum chamber.

Electrostatic PIC with adaptive Cartesian mesh

We describe an initial implementation of an electrostatic Particle-in-Cell (ES-PIC) module with adaptive Cartesian mesh in our Unified Flow Solver framework. Challenges of PIC method with cell-based adaptive mesh refinement (AMR) are related to a decrease of the particle-per-cell number in the refined cells with a corresponding increase of the numerical noise. The developed ES-PIC solver is validated for capacitively coupled plasma, its AMR capabilities are demonstrated for simulations of streamer development during high-pressure gas breakdown. It is shown that cell-based AMR provides a convenient particle management algorithm for exponential multiplications of electrons and ions in the ionization events.

A chemical reaction network solver for the astrophysics code NIRVANA

Context. Chemistry often plays an important role in astrophysical gases. It regulates thermal properties by changing species abundances and via ionization processes. This way, time-dependent cooling mechanisms and other chemistry–related energy sources can have a profound influence on the dynamical evolution of an astrophysical system. Modeling those effects with the underlying chemical kinetics in realistic magneto-gasdynamical simulations provide the basis for a better link to observations.Aims. The present work describes the implementation of a chemical reaction network solver into the magneto-gasdynamical code NIRVANA. For this purpose a multispecies structure is installed, and a new module for evolving the rate equations of chemical kinetics is developed and coupled to the dynamical part of the code. A small chemical network for a hydrogen–helium plasma was constructed including associated thermal processes which is used in test problems.Methods. Evolving a chemical network within time-dependent simulations requires the additional solution of a set of coupled advection-reaction equations for species and gas temperature. Second-order Strang-splitting is used to separate the advection part from the reaction part. The ordinary differential equation (ODE) system representing the reaction part is solved with a fourth-order generalized Runge-Kutta method applicable for stiff systems inherent to astrochemistry.Results. A series of tests was performed in order to check the correctness of numerical and technical implementation. Tests include well-known stiff ODE problems from the mathematical literature in order to confirm accuracy properties of the solver used as well as problems combining gasdynamics and chemistry. Overall, very satisfactory results are achieved.Conclusions. The NIRVANA code is now ready to handle astrochemical processes in time-dependent simulations. An easy-to-use interface allows implementation of complex networks including thermal processes. In combination with NIRVANA’s self-gravity solver, its efficient solver for dissipation terms and its adaptive mesh refinement capability challenging astrophysical problems come into reach with the code.

GRChombo: Numerical relativity with adaptive mesh refinement

In this work, we introduce a new numerical relativity code which incorporates full adaptive mesh refinement (AMR) using block structured Berger–Rigoutsos grid generation. The code supports non-trivial ‘many-boxes-in-many-boxes’ mesh hierarchies and massive parallelism through the message passing interface. evolves the Einstein equation using the standard BSSN formalism, with an option to turn on CCZ4 constraint damping if required. The AMR capability permits the study of a range of new physics which has previously been computationally infeasible in a full 3 + 1 setting, while also significantly simplifying the process of setting up the mesh for these problems. We show that can stably and accurately evolve standard spacetimes such as binary black hole mergers and scalar collapses into black holes, demonstrate the performance characteristics of our code, and discuss various physics problems which stand to benefit from the AMR technique.

A 3D front-tracking approach for simulation of a two-phase fluid with insoluble surfactant

Surface active agents play a significant role in interfacial dynamics of multiphase systems.While the understanding of their behavior is crucial to many important practical applications, realistic mathematical modeling and computer simulation represent an extraordinary task. By employing a front-tracking method with Eulerian adaptive mesh refinement capabilities in concert with a finite volume scheme for solving an advection–diffusion equation constrained to a moving and deforming interface, the numerical challenges posed by the full three-dimensional computer simulation of transient, incompressible two-phase flows with an insoluble surfactant are efficiently and accurately tackled in the present work. The individual numerical components forming the resulting methodology are here combined and applied for the first time. Verification tests to check the accuracy and the simulation of the deformation of a droplet in simple shear flow in the presence of an insoluble surfactant are performed, the results being compared to laboratory experiments as well as to other numerical data. In all the cases considered, the methodology presents excellent conservation properties for the total surfactant mass (even to machine precision under certain circumstances).

DYNAMICAL FRAGMENTATION OF THE T PYXIDIS NOVA SHELL DURING RECURRENT ERUPTIONS

Hubble Space Telescope} images of the ejecta surrounding the nova T Pyxidis resolve the emission into more than two thousand bright knots. We simulate the dynamical evolution of the ejecta from T Pyxidis during its multiple eruptions over the last 150 years using the adaptive mesh refinement capability of the gas dynamics code Ramses. We demonstrate that the observed knots are the result of Richtmeyer-Meshkov gas dynamical instabilities (the equivalent of Rayleigh-Taylor instabilities in an accelerated medium). These instabilities are caused by the overrunning of the ejecta from the classical nova of 1866 by fast moving ejecta from the subsequent six recurrent nova outbursts. The model correctly predicts the observed expansion and dimming of the T Pyx ejecta as well as the knotty morphology. The model also predicts that deeper, high resolution imagery will show filamentary structure connecting the knots. We show reprocessed Hubble Space Telescope imagery that shows the first hints of such structure.

HIGH-RESOLUTION SIMULATIONS OF CONVECTION PRECEDING IGNITION IN TYPE Ia SUPERNOVAE USING ADAPTIVE MESH REFINEMENT

We extend our previous three-dimensional, full-star simulations of the final hours of convection preceding ignition in Type Ia supernovae to higher resolution using the adaptive mesh refinement capability of our low Mach number code, MAESTRO. We report the statistics of the ignition of the first flame at an effective 4.34 km resolution, and general flow field properties at an effective 2.17 km resolution. We find that off-center ignition is likely, with radius of 50 km most favored and a likely range of 40 to 75 km. This is consistent with our previous coarser (8.68 km resolution) simulations, implying that we have achieved sufficient resolution in our determination of likely ignition radii. The dynamics of the last few hot spots preceding ignition suggest that a multiple ignition scenario is not likely. With improved resolution, we can more clearly see the general flow pattern in the convective region, characterized by a strong outward plume with a lower speed recirculation. We show that the convective core is turbulent with a Kolmogorov spectrum and has a lower turbulent intensity and larger integral length scale than previously thought (on the order of 16 km s$^{-1}$ and 200 km, respectively), and we discuss the potential consequences for the first flames.

Towards adaptive kinetic-fluid simulations of weakly ionized plasmas

This paper describes an Adaptive Mesh and Algorithm Refinement (AMAR) methodology for multi-scale simulations of gas flows and the challenges associated with extending this methodology for simulations of weakly ionized plasmas. The AMAR method combines Adaptive Mesh Refinement (AMR) with automatic selection of kinetic or continuum solvers in different parts of computational domains. We first review the discrete velocity method for solving Boltzmann and Wang Chang–Uhlenbeck kinetic equations for rarefied gases. Then, peculiarities of AMR implementation with octree Cartesian mesh are discussed. A Unified Flow Solver (UFS) uses AMAR method with adaptive Cartesian mesh to dynamically introduce kinetic patches for multi-scale simulations of gas flows. We describe fluid plasma models with AMR capabilities and illustrate how physical models affect simulation results for gas discharges, especially in the areas where electron kinetics plays an important role. We introduce Eulerian solvers for plasma kinetic equations and illustrate the concept of adaptive mesh in velocity space. Specifics of electron kinetics in collisional plasmas are described focusing on deterministic methods of solving kinetic equations for electrons under different conditions. We illustrate the appearance of distinct groups of electrons in the cathode region of DC discharges and discuss the physical models appropriate for each group. These kinetic models are currently being incorporated into AMAR methodology for multi-scale plasma simulations.

An efficient block parallel AMR method for two phase interfacial flow simulations

The direct numerical simulation of two phase interfacial flows can be computationally challenging, as the strong resolution needed to follow the deformations of the interface leads to a lot of time spent solving the whole computation domain. Efficient solution of such problems requires an adaptive mesh refinement capability to concentrate computational effort where it is most needed. In this paper a parallel adaptive algorithm to solve incompressible two-phase flows with surface tension is presented: the AMR is handled with the help of the PARAMESH package. The free interface between fluids is tracked via Level Set approach; the jump conditions at the interface for pressure and velocity are imposed by the Ghost Fluid method. A multigrid preconditioned BiCG-stab solver adapted to the AMR data structure has been developed to allow high density ratio computations (up to 1:1000). Special treatment has been done at the refinement jumps to maintain the fine mesh accuracy. Computational results are compared in different test cases with analytical solutions or literature, and show very good agreement with the references. The effectiveness of PARAMESH parallelization has been quite well maintained, as shown in the strong and weak scaling tests. Speed-up capabilities of the AMR are demonstrated.