High-dimensional PDEs Research Articles

Error analysis of kernel/GP methods for nonlinear and parametric PDEs

We introduce a priori Sobolev-space error estimates for the solution of arbitrary nonlinear, and possibly parametric, PDEs that are defined in the strong sense, using Gaussian process and kernel based methods. The primary assumptions are: (1) a continuous embedding of the reproducing kernel Hilbert space of the kernel into a Sobolev space of sufficient regularity; and (2) the stability of the differential operator and the solution map of the PDE between corresponding Sobolev spaces. The proof is articulated around Sobolev norm error estimates for kernel interpolants and relies on the minimizing norm property of the solution. The error estimates demonstrate dimension-benign convergence rates if the solution space of the PDE is smooth enough. We illustrate these points with applications to high-dimensional nonlinear elliptic PDEs and parametric PDEs. Although some recent machine learning methods have been presented as breaking the curse of dimensionality in solving high-dimensional PDEs, our analysis suggests a more nuanced picture: there is a trade-off between the regularity of the solution and the presence of the curse of dimensionality. Therefore, our results are in line with the understanding that the curse is absent when the solution is regular enough.

Mini-Workshop: Nonlinear Approximation of High-dimensional Functions in Scientific Computing

Approximation techniques for high dimensional PDEs are crucial for contemporary scientific computing tasks and gained momentum in recent years due to the renewed interest in neural networks. It seems that especially nonlinear parametrizations will play an essential role in efficient and tractable approximations of high dimensional problems. We held a mini-workshop on the relation and possible synergy of neural networks and tensor product approximation. To reliably evaluate the prospect of different numerical experiments, the traditional talks were accompanied by live coding sessions.

Adaptive sampling physics-informed neural network method for high-order rogue waves and parameters discovery of the (2 + 1)-dimensional CHKP equation.

Rogue waves are important physical phenomena, which have wide applications in nonlinear optics, hydrodynamics, Bose-Einstein condensates, and oceanic and atmospheric dynamics. We find that when using the original PINNs to study rogue waves of high dimensional PDEs, the prediction performance will become very poor, especially for high-order rogue waves due to that the randomness of selection of sample points makes insufficient use of the physical information describing the local sharp regions of rogue waves. In this paper, we propose an adaptive sampling physics-informed neural network method (ASPINN), which renders the points in local sharp regions to be selected sufficiently by a new adaptive search algorithm to lead to a prefect prediction performance. To valid the performance of our method, the (2+1)-dimensional CHKP equation is taken as an illustrative example. Experimental results reveal that the original PINNs can hardly be able to predict dynamical behaviors of the high-order rogue waves for the CHKP equation, but the ASPINN method can not only predict dynamical behaviors of these high-order rogue waves, but also greatly improve the prediction efficiency and accuracy to four orders of magnitude. Then, the data-driven inverse problem for the CHKP equation with different levels of corrupted noise is studied to show that the ASPINN method has good robustness. Moreover, some main factors affecting the neural network performance are discussed in detail, including the size of training data, the number of layers of the neural network, and the number of neurons per layer.

Tensor rank reduction via coordinate flows

Recently, there has been a growing interest in efficient numerical algorithms based on tensor networks and low-rank techniques to approximate high-dimensional functions and solutions to high-dimensional PDEs. In this paper, we propose a new tensor rank reduction method based on coordinate transformations that can greatly increase the efficiency of high-dimensional tensor approximation algorithms. The idea is simple: given a multivariate function, determine a coordinate transformation so that the function in the new coordinate system has smaller tensor rank. We restrict our analysis to linear coordinate transformations, which gives rise to a new class of functions that we refer to as tensor ridge functions. Leveraging Riemannian gradient descent on matrix manifolds we develop an algorithm that determines a quasi-optimal linear coordinate transformation for tensor rank reduction. The results we present for rank reduction via linear coordinate transformations open the possibility for generalizations to larger classes of nonlinear transformations.

Deep-Time Neural Networks: An Efficient Approach for Solving High-Dimensional PDEs

Deep-Time Neural Networks: An Efficient Approach for Solving High-Dimensional PDEs

XVA in a multi-currency setting with stochastic foreign exchange rates

In the present article we address the modelling and the numerical computation of the total value adjustment for European options in a multi-currency setting when the foreign exchange rates between the different involved currencies are assumed to be stochastic. Thus, we extend to a more realistic approach a previous work where constant exchange rates have been considered. New models are formulated both in terms of linear and nonlinear PDEs and expectations, the hedging arguments requiring the additional consideration of the exposure to foreign exchange risk. For the nonlinear models, Picard iteration methods are applied to the formulation in terms of expectations and compared with multilevel Picard iteration methods. In this way, we avoid the curse of dimensionality associated to the use of deterministic numerical methods (such as finite differences or finite element methods) for solving high dimensional PDEs. Some examples of option pricing problems illustrate the performance of the proposed models and numerical methods.

A test of backward stochastic differential equations solver for solving semilinear parabolic differential equations in 1D and 2D

Backward stochastic differential equation solver was first introduced by Han et al in 2017. A semilinear parabolic partial differential equation is converted into a stochastic differential equation, and then solved by the backward stochastic differential equation (BSDE) solver. The BSDE solver uses the backward Euler scheme for time discretization and stochastic process and neural network to learn the derivative functions at each time step. The algorithm is ideal for solving high-dimensional PDEs, especially in dimensions higher than 3-dimension problems, whereas the traditional numerical techniques fail to produce any simulations. We modified the BSDE solver so that is works for one-dimensional problems as well. The focus of this paper is to understand how the BSDE solver works in comparison with the traditional numerical techniques in low dimensional spaces (1D and 2D). We examined the BSDE solver in terms of accuracy, efficiency and convergence. Through five classical differential equations, we discovered that the solver works for low dimensional spaces, as accurate as it can be in high dimensional spaces. It is more accurate than the radial basis function collocation method reported in literature and the results by the Picard method. However, the BSDE solver is time consuming. This however, can be solved by parallel computing if needed.

Adaptive Integration of Nonlinear Evolution Equations on Tensor Manifolds

We develop new adaptive algorithms for temporal integration of nonlinear evolution equations on tensor manifolds. These algorithms, which we call step-truncation methods, are based on performing one time step with a conventional time-stepping scheme, followed by a truncation operation onto a tensor manifold. By selecting the rank of the tensor manifold adaptively to satisfy stability and accuracy requirements, we prove convergence of a wide range of step-truncation methods, including explicit one-step and multi-step methods. These methods are very easy to implement as they rely only on arithmetic operations between tensors, which can be performed by efficient and scalable parallel algorithms. Adaptive step-truncation methods can be used to compute numerical solutions of high-dimensional PDEs, which, have become central to many new areas of application such optimal mass transport, random dynamical systems, and mean field optimal control. Numerical applications are presented and discussed for a Fokker-Planck equation with spatially dependent drift on a flat torus of dimension two and four.

RBF approximation of three dimensional PDEs using tensor Krylov subspace methods

In this paper, we propose different algorithms for the solution of a tensor linear discrete ill-posed problem arising in the application of the meshless method for solving PDEs in three-dimensional space using multiquadric radial basis functions. It is well known that the truncated singular value decomposition (TSVD) is the most common effective solver for ill-conditioned systems, but unfortunately the operation count for solving a linear system with the TSVD is computationally expensive for large-scale matrices. In the present work, we propose algorithms based on the use of the well known Einstein product for two tensors to define the tensor global Arnoldi and the tensor Gloub Kahan bidiagonalization algorithms. Using the so-called Tikhonov regularization technique, we will be able to provide computable approximate regularized solutions in a few iterations. The tensor formulation will allow us to develop an RBF approximation for high dimensional PDEs based on Hierarchical tensors and Adaptive Cross Approximation, which in turn reduces significantly the storage and computational costs, while the accuracy of the method is preserved. The performance of the proposed methods is illustrated with a variety of benchmark examples and large-scale industrial applications with high degrees of freedom.

Optimally weighted loss functions for solving PDEs with Neural Networks

Recent works have shown that deep neural networks can be employed to solve partial differential equations, giving rise to the framework of physics informed neural networks (Raissi et al., 2007). We introduce a generalization for these methods that manifests as a scaling parameter which balances the relative importance of the different constraints imposed by partial differential equations. A mathematical motivation of these generalized methods is provided, which shows that for linear and well-posed partial differential equations, the functional form is convex. We then derive a choice for the scaling parameter that is optimal with respect to a measure of relative error. Because this optimal choice relies on having full knowledge of analytical solutions, we also propose a heuristic method to approximate this optimal choice. The proposed methods are compared numerically to the original methods on a variety of model partial differential equations, with the number of data points being updated adaptively. For several problems, including high-dimensional PDEs the proposed methods are shown to significantly enhance accuracy.

Solving high-dimensional Hamilton\u2013Jacobi\u2013Bellman PDEs using neural networks: perspectives from the theory of controlled diffusions and measures on path space

Optimal control of diffusion processes is intimately connected to the problem of solving certain Hamilton–Jacobi–Bellman equations. Building on recent machine learning inspired approaches towards high-dimensional PDEs, we investigate the potential of iterative diffusion optimisation techniques, in particular considering applications in importance sampling and rare event simulation, and focusing on problems without diffusion control, with linearly controlled drift and running costs that depend quadratically on the control. More generally, our methods apply to nonlinear parabolic PDEs with a certain shift invariance. The choice of an appropriate loss function being a central element in the algorithmic design, we develop a principled framework based on divergences between path measures, encompassing various existing methods. Motivated by connections to forward-backward SDEs, we propose and study the novel log-variance divergence, showing favourable properties of corresponding Monte Carlo estimators. The promise of the developed approach is exemplified by a range of high-dimensional and metastable numerical examples.

A New Technique for Preserving Conservation Laws

This paper introduces a new symbolic-numeric strategy for finding semidiscretizations of a given PDE that preserve multiple local conservation laws. We prove that for one spatial dimension, various one-step time integrators from the literature preserve fully discrete local conservation laws whose densities are either quadratic or a Hamiltonian. The approach generalizes to time integrators with more steps and conservation laws of other kinds; higher-dimensional PDEs can be treated by iterating the new strategy. We use the Boussinesq equation as a benchmark and introduce new families of schemes of order two and four that preserve three conservation laws. We show that the new technique is practicable for PDEs with three dependent variables, introducing as an example new families of second-order schemes for the potential Kadomtsev–Petviashvili equation.

DNN Expression Rate Analysis of High-Dimensional PDEs: Application to Option Pricing

We analyze approximation rates by deep ReLU networks of a class of multivariate solutions of Kolmogorov equations which arise in option pricing. Key technical devices are deep ReLU architectures capable of efficiently approximating tensor products. Combining this with results concerning the approximation of well-behaved (i.e., fulfilling some smoothness properties) univariate functions, this provides insights into rates of deep ReLU approximation of multivariate functions with tensor structures. We apply this in particular to the model problem given by the price of a European maximum option on a basket of d assets within the Black–Scholes model for European maximum option pricing. We prove that the solution to the d-variate option pricing problem can be approximated up to an varepsilon -error by a deep ReLU network with depth {mathcal {O}}big (ln (d)ln (varepsilon ^{-1})+ln (d)^2big ) and {mathcal {O}}big (d^{2+frac{1}{n}}varepsilon ^{-frac{1}{n}}big ) nonzero weights, where nin {mathbb {N}} is arbitrary (with the constant implied in {mathcal {O}}(cdot ) depending on n). The techniques developed in the constructive proof are of independent interest in the analysis of the expressive power of deep neural networks for solution manifolds of PDEs in high dimension.

A trustable shape parameter in the kernel-based collocation method with application to pricing financial options

In this paper, we focus on the kernel-based solution of high-dimensional elliptic PDEs and propose an efficient algorithm to compute a trustable value of the shape parameter for a given positive definite kernel. The proposed strategy is based on the norm-minimal generalized interpolation problem coming from the optimality of radial basis function interpolation. Also, the effectiveness of the suggested algorithm is assessed by solving parabolic partial integro-differential equations arising in option pricing theory when the dynamic of the underlying asset is driven by a Lévy process. The performance of the proposed algorithm is evaluated by some high-dimensional PDEs on regular and irregular computational domains. Indeed, the numerical results obtained with European put options under Merton’s model show the trustability of the suggested optimal shape parameter.

Deep Splitting Method for Parabolic PDEs

In this paper we introduce a numerical method for nonlinear parabolic PDEs that combines operator splitting with deep learning. It divides the PDE approximation problem into a sequence of separate learning problems. Since the computational graph for each of the subproblems is comparatively small, the approach can handle extremely high-dimensional PDEs. We test the method on different examples from physics, stochastic control and mathematical finance. In all cases, it yields very good results in up to 10,000 dimensions with short run times.

Mesh-free stochastic algorithms for systems of drift–diffusion–reaction equations and anisotropic diffusion flux calculations

We suggest in this paper two new random walk based stochastic algorithms for solving high-dimensional PDEs for domains with complicated geometrical structure. The first one, a Random Walk on Spheres (RWS) algorithm is developed for solving systems of coupled drift–diffusion–reaction equations where the random walk is living both on randomly sampled spheres and inside the relevant balls. The second method suggested solves transient anisotropic diffusion equations, where the random walk is carried out on random rectangular parallelepipeds inside the domain. The two methods are mesh-free both in space and time, and are well applied to solve high-dimensional problems with complicated domains. The algorithms are based on tracking the trajectories of the diffusing particles exactly in accordance with the probabilistic distributions derived from the explicit representation of the relevant Green functions for a sphere and a parallelepiped. They can be conveniently used not only for the solutions, but also for a direct calculation of fluxes to any part of the boundary without calculating the whole solution in the domain. Applications to exciton transport in semiconductors and related cathodoluminescence imaging of a set of randomly distributed threading dislocations are presented.

Efficient calculation of heterogeneous non-equilibrium statistics in coupled firing-rate models

Understanding nervous system function requires careful study of transient (non-equilibrium) neural response to rapidly changing, noisy input from the outside world. Such neural response results from dynamic interactions among multiple, heterogeneous brain regions. Realistic modeling of these large networks requires enormous computational resources, especially when high-dimensional parameter spaces are considered. By assuming quasi-steady-state activity, one can neglect the complex temporal dynamics; however, in many cases the quasi-steady-state assumption fails. Here, we develop a new reduction method for a general heterogeneous firing-rate model receiving background correlated noisy inputs that accurately handles highly non-equilibrium statistics and interactions of heterogeneous cells. Our method involves solving an efficient set of nonlinear ODEs, rather than time-consuming Monte Carlo simulations or high-dimensional PDEs, and it captures the entire set of first and second order statistics while allowing significant heterogeneity in all model parameters.

Moving mesh method for simulating high-dimensional time dependent PDEs with fast propagating shock waves

When solutions of the high dimensional PDE involve fast propagating shock waves, existing variational moving mesh methods are rather unstable and time consuming. To circumvent the problem, we improve the existing variational method by adding one term which adapts the mesh to the dynamic variations of the PDEs during time iterations. This is an extension of the moving mesh strategy in (Gao, 2018) for one dimensional (1-D) problems. We prove the stability and convergence of the present moving mesh equation for arbitrary initial mesh. Consequently, by enlarging the time step sizes of mesh generation, the computational time is shortened without causing instability. The moving mesh equation and the PDE are simulated step by step applying the alternate finite difference method. Both theoretical analysis and numerical results imply that the method is highly effective and robust when simulating high dimensional time dependent PDEs with fast moving shock waves.

DGM: A deep learning algorithm for solving partial differential equations

High-dimensional PDEs have been a longstanding computational challenge. We propose to solve high-dimensional PDEs by approximating the solution with a deep neural network which is trained to satisfy the differential operator, initial condition, and boundary conditions. Our algorithm is meshfree, which is key since meshes become infeasible in higher dimensions. Instead of forming a mesh, the neural network is trained on batches of randomly sampled time and space points. The algorithm is tested on a class of high-dimensional free boundary PDEs, which we are able to accurately solve in up to 200 dimensions. The algorithm is also tested on a high-dimensional Hamilton–Jacobi–Bellman PDE and Burgers' equation. The deep learning algorithm approximates the general solution to the Burgers' equation for a continuum of different boundary conditions and physical conditions (which can be viewed as a high-dimensional space). We call the algorithm a “Deep Galerkin Method (DGM)” since it is similar in spirit to Galerkin methods, with the solution approximated by a neural network instead of a linear combination of basis functions. In addition, we prove a theorem regarding the approximation power of neural networks for a class of quasilinear parabolic PDEs.

Least-Squares Proper Generalized Decompositions for Weakly Coercive Elliptic Problems

Proper generalized decompositions (PGDs) are a family of methods for efficiently solving high-dimensional PDEs, which seek to find a low-rank approximation to the solution of the PDE a priori. Convergence of PGD algorithms can only be proven for problems which are continuous, symmetric, and strongly coercive. In the particular case of problems which are only weakly coercive we have the additional issue that weak coercivity estimates are not guaranteed to be inherited by the low-rank PGD approximation. This can cause stability issues when employing a Galerkin PGD approximation of weakly coercive problems. In this paper we propose the use of PGD algorithms based on least-squares formulations which always lead to symmetric and strongly coercive problems and hence provide stable and provably convergent algorithms. Taking the Stokes problem as a prototypical example of a weakly coercive problem, we develop and compare rigorous least-squares PGD algorithms based on continuous least-squares estimates for two dif...