Adaptive Mesh Refinement Framework Research Articles

Early experiences on the OLCF Frontier system with AthenaPK and Parthenon‐Hydro

SummaryThe Oak Ridge Leadership Computing Facility (OLCF) has been preparing the nation's first exascale system, Frontier, for production and end users. Frontier is based on HPE Cray's new EX architecture and Slingshot interconnect and features 74 cabinets of optimized 3rd Gen AMD EPYC CPUs for HPC and AI and AMD Instinct 250X accelerators. As a part of this preparation, “real‐world” user codes have been selected to help assess the functionality, performance, and usability of the system. This article describes early experiences using the system in collaboration with the Hamburg Observatory for two selected codes, which have since been adopted in the OLCF test harness. Experiences discussed include efforts to resolve performance variability and per‐cycle slowdowns. Results are shown for a performance portable astrophysical magnetohydronamics code, AthenaPK, and a mini‐application stressing the core functionality of a performance portable block‐structured adaptive mesh refinement framework, Parthenon‐Hydro. These results show good scaling characteristics to the full system. At the largest scale, the Parthenon‐Hydro miniapp reaches a total of zone‐cycles/s on 9216 nodes (73,728 logical GPUs) at 92% weak scaling parallel efficiency (starting from a single node using a second‐order, finite‐volume method).

Investigations of the Atomization Characteristics and Mechanisms of Liquid Jets in Supersonic Crossflow

In the combustion chamber of scramjets, fuel jets interact with supersonic airflow in the form of a liquid jet in crossflow (LJIC). It is difficult to achieve adequate jet–crossflow mixing and the efficient combustion of fuel in an instant. Large eddy simulation (LES), the coupled level-set and volume of fluid (CLSVOF) method, and an adaptive mesh refinement (AMR) framework are used to simulate supersonic LJICs in this article. This way, LJIC atomization characteristics and mechanisms can be further explored and analyzed in detail. It is found that the surface waves of the liquid column exist in a two-dimensional form, including vertical and spanwise directions. Column breakup occurs when all the spanwise surface waves between adjacent vertical surface waves break up. Bow shock waves, composed of multiple connected arcuate shock waves, are dynamic and will change with the evolution of the liquid column. The vortex ring movement of supersonic LJICs, whose trends in the vertical and spanwise directions are different, is relatively complex, which is due to the complex and time-dependent shape of liquid columns.

Numerical study of liquid jet and shock wave induced by two-bubble collapse in open field

The shock waves and high-speed jets induced by the violent collapse of bubbles can cause serious damage to nearby structures. To fully understand the shock wave and jet characteristics induced by bubble-bubble interaction, we developed a diffuse-interface method for the simulation of compressible two-phase flow, which combined interface compression technique with interface sharpening technique to keep the sharpness of the shock wave and phase interface. The thermodynamically consistent five-equation model was improved using interface compression technique to maintain a constant thickness of the phase interface. To further suppress numerical dissipation, the high-order scheme WENO (Weighted Essentially Non-Oscillatory) and interface sharpening function THINC (Tangent of Hyperbola for INterface Capturing) were used for the flux reconstruction at the grid boundary to ensure that the total variation at the cell boundary was minimized (Boundary Variation Diminishing, BVD), thus the spatial reconstruction scheme, named WENO-THINC-BVD, was developed. Moreover, we extended the present method to a block-structured adaptive mesh refinement framework to improve grid resolution and save computing resources. Numerical results of several benchmark tests fully demonstrated the advantages of the present method in simulating complex compressible flows containing shock-shock and shock-interface interactions. Based on the present high-fidelity numerical methods, we investigated the shock wave and liquid jet induced by the two-bubble interaction in an open space. A phase diagram of the liquid jet propagation direction as a function of the initial bubble pressure and the inter-bubble distance was summarized. Finally, according to the bubble-bubble interaction process, an empirical formula for the collapse strength and another for the attenuation of shock waves were developed.

The Athena++ Adaptive Mesh Refinement Framework: Multigrid Solvers for Self-gravity

We describe the implementation of multigrid solvers in the Athena++ adaptive mesh refinement (AMR) framework and their application to the solution of the Poisson equation for self-gravity. The new solvers are built on top of the AMR hierarchy and TaskList framework of Athena++ for efficient parallelization. We adopt a conservative formulation for the Laplacian operator that avoids artificial accelerations at level boundaries. Periodic, fixed, and zero-gradient boundary conditions are implemented, as well as open boundary conditions based on a multipole expansion. Hybrid parallelization using both Message Passing Interface and OpenMP is adopted, and we present results of tests demonstrating the accuracy and scaling of the methods. On a uniform grid, we show that multigrid significantly outperforms methods based on fast Fourier transforms, and requires only a small fraction of the computing time required by the (highly optimized) magnetohydrodynamic solver in Athena++. As a demonstration of the capabilities of the methods, we present the results of a test calculation of magnetized protostellar collapse on an adaptive mesh.

A three-dimensional solver for simulating detonation on curvilinear adaptive meshes

A generic solver in a parallel Cartesian adaptive mesh refinement framework is extended to simulate detonations on three-dimensional structured curvilinear meshes. A second-order accurate finite volume method is used with grid-aligned Riemann solvers for thermally perfect gas mixtures. Detailed, multi-step chemical kinetic mechanisms are employed and numerically incorporated with a splitting approach. The adaptive mesh refinement technique is applied to a mapped mesh using modified prolongation and restriction operators. The flux along the coarse-fine interface is considered in a correction procedure to ensure the conservation of the solver. The numerical accuracy, conservation and robustness of the simulations are verified and validated with suitable benchmark tests. The new solver is then used to simulate detonation problems in non-Cartesian geometries. A simulation is conducted of the three-dimensional detonation propagation in a 90-degree pipe bend. A detonation in a round tube is also simulated in a Galilean frame of reference. Both a rectangular mode and a spinning mode are observed in the simulations. In addition, the fundamental problem of detonation wave/boundary layer interaction is studied. The results show that the new solver can simulate high-speed reactive flows efficiently by the combined use of a curvilinear mapping with mesh adaptation.

Numerical investigation of shock-induced bubble collapse dynamics and fluid–solid interactions during shock-wave lithotripsy

In this paper we investigate the bubble collapse dynamics under shock-induced loading near soft and rigid bio-materials, during shock wave lithotripsy. A novel numerical framework was developed, that employs a Diffuse Interface Method (DIM) accounting for the interaction across fluid–solid-gas interfaces. For the resolution of the extended variety of length scales, due to the dynamic and fine interfacial structures, an Adaptive Mesh Refinement (AMR) framework for unstructured grids was incorporated. This multi-material multi-scale approach aims to reduce the numerical diffusion and preserve sharp interfaces. The presented numerical framework is validated for cases of bubble dynamics, under high and low ambient pressure ratios, shock-induced collapses, and wave transmission problems across a fluid–solid interface, against theoretical and numerical results. Three different configurations of shock-induced collapse applications near a kidney stone and soft tissue have been simulated for different stand-off distances and bubble attachment configurations. The obtained results reveal the detailed collapse dynamics, jet formation, solid deformation, rebound, primary and secondary shock wave emissions, and secondary collapse that govern the near-solid collapse and penetration mechanisms. Significant correlations of the problem configuration to the overall collapse mechanisms were found, stemming from the contact angle/attachment of the bubble and from the properties of solid material. In general, bubbles with their center closer to the kidney stone surface produce more violent collapses. For the soft tissue, the bubble movement prior to the collapse is of great importance as new structures can emerge which can trap the liquid jet into induced crevices. Finally, the tissue penetration is examined for these cases and a novel tension-driven tissue injury mechanism is elucidated, emanating from the complex interaction of the bubble/tissue interaction during the secondary collapse phase of an entrapped bubble in an induced crevice with the liquid jet.

High-order Adaptive Mesh Refinement multigrid Poisson solver in any dimension

A numerical method to solve the d-dimensional Poisson equation with 2pth-order accuracy for arbitrary d and p integers is proposed in the Adaptive Mesh Refinement framework. Compact finite differences provide high-order compact stencils fitted for the AMR framework where reaching far-away neighboring points is very penalizing. Vertex-centered mesh refinement and interpolation ease the implementation of a multigrid algorithm formulated in the general case for any stencil in any dimension. Its computational costs are compared to those of other existing methods. And, in extensive numerical experiments, a sixth-order version of it in dimensions two to six and a tenth-order version in dimension three are tested.

Effects of staggered opposed hot jets on the initiation and propagation of gaseous detonation in a supersonic combustible inflow

In this paper, the detonation initiation mechanism of a supersonic combustible mixture triggered by a staggered opposing combined hot jets was performed. Two-dimensional reactive Navier–Stokes equations with a one-step Arrhenius chemistry model were solved using a structured adaptive mesh refinement framework. The results show that a high temperature and pressure region triggers a rapid detonation initiation after the jet-induced bow shock focusing. Further analysis showed that there is a large baroclinic torque behind the local detonation wave induced by the staggered hot jet, which leads to a large Richtmyer–Meshkov instability at the end of the unburned jet, and the generated periodic shedding vortex structure thereby enhances the diffusion effect in the unburned region. However, the released heat cannot support the propagation of the detonation wave. In addition, different jet intensity distribution schemes and jet spacing will change the ignition point position. It is worth noting that the distance of detonation initiation can be significantly shortened by reducing the front jet intensity while maintaining the total jet energy. Increasing the jet spacing will significantly slow down the detonation initiation process.

A MOOD-like compact high order finite volume scheme with adaptive mesh refinement

In this paper, a novel Finite Volume (FV) scheme for obtaining high order approximations of solutions of multi-dimensional hyperbolic systems of conservation laws within an Adaptive Mesh Refinement framework is proposed. It is based on a point-wise polynomial reconstruction that avoids the recalculation of reconstruction stencils and matrices whenever a mesh is refined or coarsened. It also couples both the limiting of the FV scheme and the refinement procedure, taking advantage of the Multi-dimensional Optimal Order Detection (MOOD) detection criteria. The resulting computational procedure is employed to simulate test cases of increasing difficulty using two models of Partial Differential Equations: the Euler system and the radiative M1 model, thus demonstrating its efficiency.

Parthenon—a performance portable block-structured adaptive mesh refinement framework

On the path to exascale the landscape of computer device architectures and corresponding programming models has become much more diverse. While various low-level performance portable programming models are available, support at the application level lacks behind. To address this issue, we present the performance portable block-structured adaptive mesh refinement (AMR) framework Parthenon, derived from the well-tested and widely used Athena++ astrophysical magnetohydrodynamics code, but generalized to serve as the foundation for a variety of downstream multi-physics codes. Parthenon adopts the Kokkos programming model, and provides various levels of abstractions from multidimensional variables, to packages defining and separating components, to launching of parallel compute kernels. Parthenon allocates all data in device memory to reduce data movement, supports the logical packing of variables and mesh blocks to reduce kernel launch overhead, and employs one-sided, asynchronous MPI calls to reduce communication overhead in multi-node simulations. Using a hydrodynamics miniapp, we demonstrate weak and strong scaling on various architectures including AMD and NVIDIA GPUs, Intel and AMD x86 CPUs, IBM Power9 CPUs, as well as Fujitsu A64FX CPUs. At the largest scale on Frontier (the first TOP500 exascale machine), the miniapp reaches a total of 1.7 × 1013 zone-cycles/s on 9216 nodes (73,728 logical GPUs) at [Formula: see text] weak scaling parallel efficiency (starting from a single node). In combination with being an open, collaborative project, this makes Parthenon an ideal framework to target exascale simulations in which the downstream developers can focus on their specific application rather than on the complexity of handling massively-parallel, device-accelerated AMR.

SADHANA: A Doubly Linked List-based Multidimensional Adaptive Mesh Refinement Framework for Solving Hyperbolic Conservation Laws with Application to Astrophysical Hydrodynamics and Magnetohydrodynamics

We report the development of a nested block-structured adaptive mesh framework to solve multidimensional, time-dependent hyperbolic equations encountered in astrophysics. An approach based on a tabular list is used to construct variants of Hilbert space-filling curves in an iterative fashion to maintain the connectivity of locally refined mesh configurations using a doubly linked list. Modifications are made to conventional boundaries of computational blocks to aid the adaptive mesh. We also describe a well-defined, computationally efficient data structure to hold self-similar mesh units for this purpose. The flexibility of this code is demonstrated by the performance of various Riemann solvers implemented in this computational framework.

Computation of interfacial flows using Continuous Surface Stress method with adaptive mesh refinement in a quad/octree grid structure

A new implementation of the Continuous Surface Stress (CSS) method into a quad/octree grid structure is presented, which realizes effective adaptive mesh refinement (AMR) for computation of interfacial flows with significant topology changes. The momentum conservation property of the CSS method is exactly retained. The extension of the CSS method to the AMR framework and a designed phase-based refinement criterion are described in detail. The new implementation works on 2D/3D and on axis-symmetric grids. The proposed method is shown to be accurate in the computation of the Laplace pressure. The spurious velocity is evaluated based on the benchmark cases of static droplet and translating droplet. The test case of multiple droplets colliding onto a liquid film illustrates the functionality and efficiency of the method in dealing with complex interfacial flows. Thanks to the high resolution realized by AMR, it is shown in computation of the process of binary droplet collisions that the method is capable of capturing finest interfacial structures subjected to complex topology changes.

A moving embedded boundary approach for the compressible Navier-Stokes equations in a block-structured adaptive refinement framework

A computational technique has been developed to perform compressible flow simulations involving moving boundaries using an embedded boundary approach within the block-structured adaptive mesh refinement (SAMR) framework of AMReX [1,91,92]. We leverage the SAMR capability to obtain quantitatively accurate results whilst using robust, second-order finite volume schemes. A conservative, unsplit, cut-cell approach is utilized and a ghost-cell approach is developed for computing the flux on the moving, embedded boundary faces. A third-order least-squares formulation has been developed to compute the wall velocity gradients, and was found to significantly improve the performance of the solver in terms of the quantitative comparison of surface quantities such as the skin friction coefficient. Various test cases are performed to validate the method, and compared with analytical, experimental, and other numerical results in literature. Inviscid and viscous test cases are performed that span a wide regime of flow speeds – acoustic (harmonically pulsating sphere), smooth flows (expansion fan created by a receding piston) and flows with shocks (shock-cylinder interaction, shock-wedge interaction, pitching NACA 0012 airfoil and shock-cone interaction). A closed system with moving boundaries – an oscillating piston in a cylinder, showed that the percentage error in mass within the system decreases with refinement, demonstrating that the numerical scheme is conservative with grid refinement, but is not discretely conservative. Viscous test cases involve that of a horizontally moving cylinder at Re=40, an inline oscillating cylinder at Re=100, and a transversely oscillating cylinder at Re=185. The judicious use of adaptive mesh refinement with appropriate refinement criteria to capture the regions of interest leads to well-resolved flow features, and good quantitative comparison is observed with the results available in literature.

Reacting condensed phase explosives in direct contact

In this article, we present a new formulation and an associated algorithm for the simultaneous numerical simulation of multiple condensed phase explosives in direct contact with each other, which may also be confined by (or interacting with one or more) compliant inert materials. Examples include composite rate-stick (i.e., involving two explosives in contact) problems, interaction of shock waves with chemically active particles in condensed-phase explosives, and devices such as detonators and boosters. There are several formulations that address the compliant or structural response of confiners and particles due to detonations, but the direct interaction of explosives remains a challenge for most formulations and algorithms. The proposed formulation addresses this problem by extending the conservation laws and mixture rules of an existing hybrid formulation (suitable for solving problems involving the coexistence of reactants and products in an explosive mixture and its immiscible interaction with inert materials) to model the interaction of multiple explosive mixtures. An algorithm for the solution of the resulting system of partial differential equations is presented, which includes a new robust method for the retrieval of the densities of the constituents of each explosive mixture. This is achieved by means of a multi-dimensional root-finding algorithm, which employs physical as well as mathematical considerations in order to converge to the correct solution. The algorithm is implemented in a hierarchical adaptive mesh refinement framework and validated against results from problems with known solutions. Additional case studies demonstrate that the method can simulate the interaction of detonation waves produced by military grade and commercial explosives in direct contact, each with its own distinct equation of state and reaction rate law.

An efficient geometry-adaptive mesh refinement framework and its application in the immersed boundary lattice Boltzmann method

This work presents an adaptive mesh refinement (AMR) method. This AMR is based on a pointless representation of octrees, i.e. hash table. An individual hash table is used for each level of refinement to avoid conflicts of key values. Cases for two complex geometries are presented to analyse the performance of the AMR framework with different data structures and strategies. Then, the immersed boundary-lattice Boltzmann method (IB-LBM) is implemented as an example to evaluate the performance of the proposed AMR framework in computational fluid dynamics (CFD) applications. The integrated solver is validated and its performance is analysed through several cases. Considerable reductions of more than 65% in computational time are achieved when the hash table is adopted instead of the binary tree. Moreover, hash table offers more benefits to 3D cases with larger numbers of nodes. The AMR framework presented in this study is simple to implement with the standard C++ libraries and time-efficient for computational fluid dynamics (CFD) applications, which is released as open-source software and can be used by researchers with their CFD solvers.

Simulation of free-surface flows using the Lattice Boltzmann method with the AMR technique

High resolution grids are generally preferred near the free surface to capture desired phenomena. In the present paper, an adaptive mesh refinement (AMR) framework based on the hash table data structure is introduced to the lattice Boltzmann method (LBM). The AMR refinement criterion is based on the movement of the free surface captured through interface nodes. The performance of the AMR-based LBM solver is analyzed through several benchmark cases. Results show that the proposed numerical framework can properly track the free surface with a spatial convergence rate between the first and second order. In addition, the AMR framework is of benefit to efficiently simulating free-surface flows since it requires much less nodes to achieve high resolution near the interface.

Efficient 2D acoustic wave finite-difference numerical simulation in strongly heterogeneous media using the adaptive mesh refinement technique

The finite-difference method (FDM) is one of the most popular numerical methods to simulate seismic wave propagation in complex velocity models. If a uniform grid is applied in FDM for heterogeneous models, the grid spacing is determined by the global minimum velocity to suppress dispersion and dissipation errors in the numerical scheme, resulting in spatial oversampling in higher velocity zones. Then, the small grid spacing dictates a small time step due to the stability condition of explicit numerical schemes. The spatial oversampling and reduced time step will cause unnecessarily inefficient use of memory and computational resources in simulations for strongly heterogeneous media. To overcome this problem, we have used the adaptive mesh refinement (AMR) technique in the FDM to flexibly adjust the grid spacing following velocity variations. AMR is rarely used in acoustic wave simulations with the FDM due to the increased complexity of implementation, including its data management, grid generation, and computational load balancing on high-performance computing platforms. We implement AMR for 2D acoustic wave simulation in strongly heterogeneous media based on the patch approach with FDM. The AMR grid can be automatically generated for given velocity models. To simplify the implementation, we use a well-developed AMR framework, AMReX, to carry out the complex grid management. Numerical tests determine the stability, accuracy level, and efficiency of the AMR scheme. The computation time is approximately proportional to the number of grid points, and the overhead due to the wavefield exchange and data structure is small.

A solver for simulating shock-induced combustion on curvilinear adaptive meshes

A generic solver in a structured Cartesian adaptive mesh refinement framework is extended to simulate unsteady shock-induced combustion problems on a structured curvilinear mesh. A second-order accurate finite volume method is used with a grid-aligned Riemann solver for inviscid thermally perfect gas mixtures. To solve these reactive problems, detailed chemical kinetic mechanisms are employed with a splitting approach. The prolongation and restriction operators are modified to implement the adaptive mesh refinement algorithm on a mapped mesh. The developed solver is verified with several benchmark tests and is then used to simulate unsteady shock-induced combustion. The results show that the computed stand-off distance of waves and oscillation frequencies of mass fraction of products observed at the stagnation point are in good agreement with the results from experiments.

A numerical method for self-similar solutions of ideal magnetohydrodynamics

We present a numerical method to obtain self-similar solutions of the ideal magnetohydrodynamics (MHD) equations. Under a self-similar transformation, the initial value problem (IVP) is converted into a boundary value problem (BVP) by eliminating time and transforming the system to self-similar coordinates (ξ≡x/t,η≡y/t). The ideal MHD system of equations is augmented by a generalized Lagrange multiplier (GLM) to maintain the solenoidal condition on the magnetic field. The self-similar solution to the BVP is solved using an iterative method, and implemented using the p4est adaptive mesh refinement (AMR) framework. Existing Riemann solvers (e.g., Roe, HLLD etc.) can be modified in a relatively straightforward manner and used in the present method. Numerical tests illustrate that the present self-similar solution to the BVP exhibits sharper discontinuities than the corresponding one solved by the IVP. We compare and contrast the IVP and BVP solutions in several one dimensional shock-tube test problems and two dimensional test cases include shock wave refraction at a contact discontinuity, reflection at a solid wall, and shock wave diffraction over a right angle corner.

Exascale models of stellar explosions: Quintessential multi-physics simulation

The ExaStar project aims to deliver an efficient, versatile, and portable software ecosystem for multi-physics astrophysics simulations run on exascale machines. The code suite is a component-based multi-physics toolkit, built on the capabilities of current simulation codes (in particular Flash-X and Castro), and based on the massively parallel adaptive mesh refinement framework AMReX. It includes modules for hydrodynamics, advanced radiation transport, thermonuclear kinetics, and nuclear microphysics. The code will reach exascale efficiency by building upon current multi- and many-core packages integrated into an orchestration system that uses a combination of configuration tools, code translators, and a domain-specific asynchronous runtime to manage performance across a range of platform architectures. The target science includes multi-physics simulations of astrophysical explosions (such as supernovae and neutron star mergers) to understand the cosmic origin of the elements and the fundamental physics of matter and neutrinos under extreme conditions.