Elliptic Solver Research Articles

Vortex-p: A Helmholtz-Hodge and Reynolds decomposition algorithm for particle-based simulations

Astrophysical turbulent flows display an intrinsically multi-scale nature, making their numerical simulation and the subsequent analyses of simulated data a complex problem. In particular, two fundamental steps in the study of turbulent velocity fields are the Helmholtz-Hodge decomposition (compressive+solenoidal; HHD) and the Reynolds decomposition (bulk+turbulent; RD). These problems are relatively simple to perform numerically for uniformly-sampled data, such as the one emerging from Eulerian, fix-grid simulations; but their computation is remarkably more complex in the case of non-uniformly sampled data, such as the one stemming from particle-based or meshless simulations. In this paper, we describe, implement and test vortex-p, a publicly available tool evolved from the vortex code, to perform both these decompositions upon the velocity fields of particle-based simulations, either from smoothed particle hydrodynamics (SPH), moving-mesh or meshless codes. The algorithm relies on the creation of an ad-hoc adaptive mesh refinement (AMR) set of grids, on which the input velocity field is represented. HHD is then addressed by means of elliptic solvers, while for the RD we adapt an iterative, multi-scale filter. We perform a series of idealised tests to assess the accuracy, convergence and scaling of the code. Finally, we present some applications of the code to various SPH and meshless finite-mass (MFM) simulations of galaxy clusters performed with OpenGadget3, with different resolutions and physics, to showcase the capabilities of the code.

Sparse Recovery of Elliptic Solvers from Matrix-Vector Products

Sparse Recovery of Elliptic Solvers from Matrix-Vector Products

Exascale Computational Fluid Dynamics in Heterogeneous Systems

Abstract Exascale computing has extended the reach of resolved flow simulations in complex, heterogeneous systems far beyond conventional computational fluid dynamics capabilities. As a result, unprecedented pore and microscale resolution have been achieved in domains that have been traditionally modeled by, and limited to, continuum, effective medium approaches. By making use of computational resources on the new exascale supercomputer, Frontier, at the Oak Ridge Leadership Computing Facility, we performed flow simulations that have pushed the limits of domain-to-resolution ratios by several orders of magnitude for heterogeneous media. Our approach is an incompressible, Navier–Stokes CFD solver based on adaptive, embedded boundary (EB) methods supported by the Chombo software framework for applied partial differential equations (PDEs). The computational workhorse in the CFD application code is an elliptic solver framework in Chombo for pressure-Poisson and viscous, Helmholtz terms that leverages a PETSc-hypre software interface tuned for accelerator-based platforms. We demonstrate scalability of the approach by replicating a unit cylinder packed with microspheres to achieve over 400 × 109 degrees-of-freedom simulated. These simulations model domain lengths of over 20 meters with channel volumes of over 400 cm3 and containing millions of packed spheres with 20 micron grid resolution, challenging current understanding of what it means to be a representative elementary volume (REV) of the continuum scale in heterogeneous media. We also simulate a range of Reynolds numbers to demonstrate wide applicability and robustness of the approach.

Solving the pulsar equation using physics-informed neural networks

ABSTRACT In this study, Physics-Informed Neural Networks (PINNs) are skilfully applied to explore a diverse range of pulsar magnetospheric models, specifically focusing on axisymmetric cases. The study successfully reproduced various axisymmetric models found in the literature, including those with non-dipolar configurations, while effectively characterizing current sheet features. Energy losses in all studied models were found to exhibit reasonable similarity, differing by no more than a factor of three from the classical dipole case. This research lays the groundwork for a reliable elliptic Partial Differential Equation solver tailored for astrophysical problems. Based on these findings, we foresee that the utilization of PINNs will become the most efficient approach in modelling three-dimensional magnetospheres. This methodology shows significant potential and facilitates an effortless generalization, contributing to the advancement of our understanding of pulsar magnetospheres.

A discontinuous Galerkin discretization of elliptic problems with improved convergence properties using summation by parts operators

Nishikawa (2007) proposed to reformulate the classical Poisson equation as a steady state problem for a linear hyperbolic system. This results in optimal error estimates for both the solution of the elliptic equation and its gradient. However, it prevents the application of well-known solvers for elliptic problems. We show connections to a discontinuous Galerkin (DG) method analyzed by Cockburn, Guzmán, and Wang (2009) that is very difficult to implement in general. Next, we demonstrate how this method can be implemented efficiently using summation by parts (SBP) operators, in particular in the context of SBP DG methods such as the DG spectral element method (DGSEM). The resulting scheme combines nice properties of both the hyperbolic and the elliptic point of view, in particular a high order of convergence of the gradients, which is one order higher than what one would usually expect from DG methods for elliptic problems.

Modelling force-free neutron star magnetospheres using physics-informed neural networks

ABSTRACT Using physics-informed neural networks (PINNs) to solve a specific boundary value problem is becoming more popular as an alternative to traditional methods. However, depending on the specific problem, they could be computationally expensive and potentially less accurate. The functionality of PINNs for real-world physical problems can significantly improve if they become more flexible and adaptable. To address this, our work explores the idea of training a PINN for general boundary conditions and source terms expressed through a limited number of coefficients, introduced as additional inputs in the network. Although this process increases the dimensionality and is computationally costly, using the trained network to evaluate new general solutions is much faster. Our results indicate that PINN solutions are relatively accurate, reliable, and well behaved. We applied this idea to the astrophysical scenario of the magnetic field evolution in the interior of a neutron star connected to a force-free magnetosphere. Solving this problem through a global simulation in the entire domain is expensive due to the elliptic solver’s needs for the exterior solution. The computational cost with a PINN was more than an order of magnitude lower than the similar case solved with a finite difference scheme, arguably at the cost of accuracy. These results pave the way for the future extension to three-dimensional of this (or a similar) problem, where generalized boundary conditions are very costly to implement.

On the impact of differential diffusion between soot and gas phase species in turbulent flames

The molecular diffusivities of larger PAHs and soot particles approach zero leading to differential diffusion with gas-phase species. The present work systematically quantifies the impact on soot moments, soot related statistical correlations and Particle Size Distributions (PSDs) using a fully coupled transported joint probability density function (JPDF) method featuring a 78-dimensional joint-scalar space, including enthalpy, gas phase species with the PSD discretised using 62 size classes via a mass and number density preserving sectional method. Differential diffusion of soot (DDS) is treated via a gradual decline of diffusivity among soot sections maintaining realisability and the expected exponential decay of variance. The solution of the flow field features a time-dependent second moment closure and an elliptic solver. The turbulent non-premixed Sandia C2H4 flame from the International Sooting Flame (ISF) data base was selected as a target along with the KAUST (C2H4/N2) variant of the same flame. Results show that reduced soot diffusion leads to a significant increase in the soot volume fraction RMS and that the correlation coefficient between soot volume fraction and temperature is further reduced in a particle-size-dependent manner. Similar observations are made for correlations between the soot volume fraction and the mass fractions of gas-phase species such as CO, OH, H and C2H2. The results suggest that computational methods that presume explicit (e.g. flame structure related) correlations between such scalars and with soot face leading order modeling challenges. It is also shown that the correlation between CO and soot increases due to oxidation of soot and that DDS leads to a modest downstream shift of PSDs towards larger particles.

Editorial: Tensor numerical methods and their application in scientific computing and data science

Editorial: Tensor numerical methods and their application in scientific computing and data science

Low dispersion finite volume/element discretization of the enhanced Green–Naghdi equations for wave propagation, breaking and runup on unstructured meshes

We study a hybrid approach combining a finite volume (FV) and a finite element (FE) method to solve a fully-nonlinear and weakly-dispersive depth averaged wave propagation model. The FV method is used to solve the underlying hyperbolic shallow water system, while a standard P1 finite element method is used to solve the elliptic system associated to the dispersive correction. We study the impact of several numerical aspects: the impact of the reconstruction used in the hyperbolic phase; the representation of the FV data in the FE method used in the elliptic phase and their impact on the theoretical accuracy of the method; the well-posedness of the overall method. For the first element we proposed a systematic implementation of an iterative reconstruction providing on arbitrary meshes up to third order solutions, full second order first derivatives, as well as a consistent approximation of the second derivatives. These properties are exploited to improve the assembly of the elliptic solver, showing dramatic improvement of the finale accuracy, if the FV representation is correctly accounted for. Concerning the elliptic step, the original problem is usually better suited for an approximation in H(div) spaces. However, it has been shown that perturbed problems involving similar operators with a small Laplace perturbation are well behaved in H1. We show, based on both heuristic and strong numerical evidence, that numerical dissipation plays a major role in stabilizing the coupled method, and not only providing convergent results, but also providing the expected convergence rates. Finally, the full mode, coupling a wave breaking closure previously developed by the authors, is thoroughly tested on standard benchmarks using unstructured grids with sizes comparable or coarser than those usually proposed in literature.

Mixed-Precision for Linear Solvers in Global Geophysical Flows.

Semi‐implicit (SI) time‐stepping schemes for atmosphere and ocean models require elliptic solvers that work efficiently on modern supercomputers. This paper reports our study of the potential computational savings when using mixed precision arithmetic in the elliptic solvers. Precision levels as low as half (16 bits) are used and a detailed evaluation of the impact of reduced precision on the solver convergence and the solution quality is performed. This study is conducted in the context of a novel SI shallow‐water model on the sphere, purposely designed to mimic numerical intricacies of modern all‐scale weather and climate (W&C) models. The governing algorithm of the shallow‐water model is based on the non‐oscillatory MPDATA methods for geophysical flows, whereas the resulting elliptic problem employs a strongly preconditioned non‐symmetric Krylov‐subspace Generalized Conjugated‐Residual (GCR) solver, proven in advanced atmospheric applications. The classical longitude/latitude grid is deliberately chosen to retain the stiffness of global W&C models. The analysis of the precision reduction is done on a software level, using an emulator, whereas the performance is measured on actual reduced precision hardware. The reduced‐precision experiments are conducted for established dynamical‐core test‐cases, like the Rossby‐Haurwitz wavenumber 4 and a zonal orographic flow. The study shows that selected key components of the elliptic solver, most prominently the preconditioning and the application of the linear operator, can be performed at the level of half precision. For these components, the use of half precision is found to yield a speed‐up of a factor 4 compared to double precision for a wide range of problem sizes.

Fast hyperbolic relaxation elliptic solver for numerical relativity: Conformally flat, binary puncture initial data

We introduce nrpyelliptic, an elliptic solver for numerical relativity (NR) built within the nrpy+ framework. As its first application, nrpyelliptic sets up conformally flat, binary black hole (BBH) puncture initial data (ID) on a single numerical domain, similar to the widely used twopunctures code. Unlike twopunctures, nrpyelliptic employs a hyperbolic relaxation scheme, whereby arbitrary elliptic partial differential equations (PDEs) are trivially transformed into a hyperbolic system of PDEs. As consumers of NR ID generally already possess expertise in solving hyperbolic PDEs, they will generally find nrpyelliptic easier to tweak and extend than other NR elliptic solvers. When evolved forward in (pseudo)time, the hyperbolic system exponentially reaches a steady state that solves the elliptic PDEs. Notably nrpyelliptic accelerates the relaxation waves, which makes it many orders of magnitude faster than the usual constant wave speed approach. While it is still $\ensuremath{\sim}12\mathrm{x}$ slower than twopunctures at setting up full-3D BBH ID, nrpyelliptic requires only $\ensuremath{\approx}0.3%$ of the runtime for a full BBH simulation in the einstein toolkit. Future work will focus on improving performance and generating other types of ID, such as binary neutron stars.

A scalable elliptic solver with task-based parallelism for the SpECTRE numerical relativity code

Elliptic partial differential equations must be solved numerically for many problems in numerical relativity, such as initial data for every simulation of merging black holes and neutron stars. Existing elliptic solvers can take multiple days to solve these problems at high resolution and when matter is involved, because they are either hard to parallelize or require a large amount of computational resources. Here we present a new solver for linear and nonlinear elliptic problems that is designed to scale with resolution and to parallelize on computing clusters. To achieve this we employ a discontinuous Galerkin discretization, an iterative multigrid-Schwarz preconditioned Newton-Krylov algorithm, and a task-based parallelism paradigm. To accelerate convergence of the elliptic solver we have developed novel subdomain-preconditioning techniques. We find that our multigrid-Schwarz preconditioned elliptic solves achieve iteration counts that are independent of resolution, and our task-based parallel programs scale over 200 million degrees of freedom to at least a few thousand cores. Our new code solves a classic initial data problem for binary black holes faster than the spectral code SpEC when distributed to only eight cores, and in a fraction of the time on more cores. It is publicly accessible in the next-generation SpECTRE numerical relativity code. Our results pave the way for highly parallel elliptic solves in numerical relativity and beyond.

Solving 3D magnetohydrostatics with RBF-FD: Applications to the solar corona

We present a novel magnetohydrostatic numerical model that solves directly for the force-balanced magnetic field in the solar corona. This model is constructed with Radial Basis Function Finite Differences (RBF-FD), specifically 3D polyharmonic splines plus polynomials, as the core discretization. This set of PDEs is particularly difficult to solve since in the limit of the forcing going to zero it becomes ill-posed with a multitude of solutions. For the forcing equal to zero there are no numerically tractable solutions. For finite forcing, the ability to converge onto a physically viable solution is delicate as will be demonstrated. The static force-balance equations are of a hyperbolic nature, in that information of the magnetic field travels along characteristic surfaces, yet they require an elliptic type solver approach for a sparse overdetermined ill-conditioned system. As an example, we reconstruct a highly nonlinear analytic model designed to represent long-lived magnetic structures observed in the solar corona.

Gmunu: paralleled, grid-adaptive, general-relativistic magnetohydrodynamics in curvilinear geometries in dynamical space–times

ABSTRACT We present an update on the General-relativistic multigrid numerical (Gmunu) code, a parallelized, multidimensional curvilinear, general relativistic magnetohydrodynamics code with an efficient non-linear cell-centred multigrid elliptic solver, which is fully coupled with an efficient block-based adaptive mesh refinement module. To date, as described in this paper, Gmunu is able to solve the elliptic metric equations in the conformally flat condition approximation with the multigrid approach and the equations of ideal general-relativistic magnetohydrodynamics by means of high-resolution shock-capturing finite-volume method with reference metric formularised multidimensionally in Cartesian, cylindrical, or spherical geometries. To guarantee the absence of magnetic monopoles during the evolution, we have developed an elliptical divergence cleaning method by using the multigrid solver. In this paper, we present the methodology, full evolution equations and implementation details of Gmunu and its properties and performance in some benchmarking and challenging relativistic magnetohydrodynamics problems.

New public code for initial data of unequal-mass, spinning compact-object binaries

The construction of constraint-satisfying initial data is an essential element for the numerical exploration of the dynamics of compact-object binaries. While several codes have been developed over the years to compute generic quasi-equilibrium configurations of binaries comprising either two black holes, or two neutron stars, or a black hole and a neutron star, these codes are often not publicly available or they provide only a limited capability in terms of mass ratios and spins of the components in the binary. We here present a new open-source collection of spectral elliptic solvers that are capable of exploring the major parameter space of binary black holes (BBHs), binary neutron stars (BNSs), and mixed binaries of black holes and neutron stars (BHNSs). Particularly important is the ability of the spectral-solver library to handle neutron stars that are either irrotational or with an intrinsic spin angular momentum that is parallel to the orbital one. By supporting both analytic and tabulated equations of state at zero or finite temperature, the new infrastructure is particularly geared towards allowing for the construction of BHNS and BNS binaries. For the latter, we show that the new solvers are able to reach the most extreme corners in the physically plausible space of parameters, including extreme mass ratios and spin asymmetries, thus representing the most extreme BNS computed to date. Through a systematic series of examples, we demonstrate that the solvers are able to construct quasi-equilibrium and eccentricity-reduced initial data for BBHs, BNSs, and BHNSs, achieving spectral convergence in all cases. Furthermore, using such initial data, we have carried out evolutions of these systems from the inspiral to after the merger, obtaining evolutions with eccentricities $\lesssim 10^{-4}-10^{-3}$, and accurate gravitational waveforms.

Gradient flow formulation and second order numerical method for motion by mean curvature and contact line dynamics on rough surface

We study the dynamics of a droplet moving on an inclined rough surface in the absence of inertial and viscous stress effects. In this case, the dynamics of the droplet is a purely geometric motion in terms of the wetting domain and the capillary surface. Using a single graph representation, we interpret this geometric motion as a gradient flow on a manifold. We propose unconditionally stable first/second order numerical schemes to simulate this geometric motion of the droplet, which is described using motion by mean curvature coupled with moving contact lines. The schemes are based on (i) explicit moving boundaries, which decouple the dynamic updates of the contact lines and the capillary surface, (ii) an arbitrary Lagrangian–Eulerian method on moving grids and (iii) a predictor-corrector method with a nonlinear elliptic solver up to second order accuracy. For the case of quasi-static dynamics with continuous spatial variable in the numerical schemes, we prove the stability and convergence of the first/second order numerical schemes. To demonstrate the accuracy and long-time validation of the proposed schemes, several challenging computational examples – including breathing droplets, droplets on inhomogeneous rough surfaces and quasi-static Kelvin pendant droplets – are constructed and compared with exact solutions to quasi-static dynamics obtained by desingularized differential-algebraic system of equations (DAEs).

Development of the global multiscale atmosphere model: computational aspects

The global atmosphere model SL-AV is applied for operational numerical weather prediction in Russia at the time scales from days to month and also for simulation of modern climate. It is important to achieve the best wall-clock time for all of the model applications with different grid sizes. In this paper, we analyse the effect of switching computations and parallel exchanges from double to single precision in the semi-Lagrangian advection and elliptic equations solver blocks on the quality of the medium-range forecast and evaluate the change in wall-clock time. We have also optimized OpenMP parallelization in the computations of the right-hand sides of meteorological prognostic equations. The effect of these changes is studied for SL-AV model versions with different resolutions.

An adaptive time stepping algorithm for IMPES with high order polynomial extrapolation

Two-phase flows in oil reservoirs can be modeled by a coupled system of elliptic and hyperbolic partial differential equations. The transport velocity of the multiphase fluid system is related to the pressure through Darcy’s law and it is coupled to a conservation law for the saturation variable of one of the phases. A time step of the classical IMPES (IMplicit Pressure Explicit Saturation) method consists of first solving the elliptic problem for pressure and Darcy velocity, and then updating the saturation with an explicit numerical scheme for conservation laws. This method is very computationally costly, since the time-consuming elliptic solver must be invoked at time intervals defined by the stability limit of the hyperbolic solver. A popular variant is not to update the velocity at all hyperbolic time steps, but to skip a fixed number C of them, with C determined by the user. In this work we propose a more accurate and systematic procedure for time stepping in IMPES codes. The velocity is updated at all transport time steps, though the elliptic solver is only invoked every C steps. In the time steps at which the elliptic problem is not solved, the velocity is extrapolated from previously computed values with polynomials of high degree. Further, we introduce an error estimator that allows for the number C to be adaptively determined without user intervention. The algorithm was tested in several relevant benchmark problems. This allowed for the optimization of its parameters and comparisons with previous variants. The results show that the proposed algorithm is very stable, reliable and time-cost effective. It is also easily implemented in pre-existent IMPES codes.

Linear computational cost implicit solver for elliptic problems

In this paper, we use the alternating direction method for isogeometric finite elements to simulate implicit dynamics. Namely, we focus on a parabolic problem and use B-spline basis functions in space and an implicit marching method to fully discretize the problem. We introduce intermediate time steps and separate our differential operator into a summation of the blocks, acting along a particular coordinate axis in the intermediate time steps. We show that the resulting stiffness matrix can be represented as a multiplication of two (in 2D) or three (in 3D) multi-diagonal matrices, each one with B-spline basis functions along the particular axis of the spatial system of coordinates. As a result of this algebraic transformations, we get a system of linear equations that can be factorized in linear $O(N)$ computational cost in every time step of the implicit method. We use our method to simulate the heat transfer problem. We demonstrate theoretically and verify numerically that our implicit method is unconditionally stable for heat transfer problems (i.e., parabolic). We conclude our presentation with a discussion on the limitations of the method.

An algebraic multiscale solver with local Robin boundary value problems for flows in high-contrast media

The algebraic and semi-algebraic formulations of a new multiscale elliptic solver based on a non-overlapping domain decomposition method are presented. By algebraic we mean that all information needed to implement the proposed procedure can be extracted from the underlying fine grid finite volume linear system. In addition to the entries of this linear system, the proposed semi-algebraic procedure will also use the coefficients of the elliptic equation at subdomain boundaries. Initially we construct multiscale basis functions (or local solutions) subject to Robin boundary conditions. Although the implementation of Robin conditions in the formulation of the multiscale method uses the coefficients of the elliptic equation of interest, we modify it such that only entries of the finite volume linear system appear in the calculation of local solutions. A linear combination of local solutions gives a (discontinuous at subdomain boundaries) solution that is refined in a smoothing step. To this end we propose an algebraic scheme to remove discontinuities that needs a staggered (or dual) coarse grid. By iterating with the defect correction scheme, we construct an effective two-stage preconditioner that combines both local solutions and the smoothing step; the final result is obtained in a small number of iterations. Our focus in this work is the presentation of the algebraic and semi-algebraic formulations of the multiscale elliptic solver, which are carefully explained, and in verifying their accuracy and efficiency in the solution of challenging problems. We consider two- and three-dimensional problems with coefficients exhibiting high-contrast, anisotropic, and channelized structures that are solved to reveal the good properties of the proposed schemes.