Dimension-independent Convergence Rates Research Articles

Randomly Shifted Lattice Rules with Importance Sampling and Applications

In financial and statistical computations, calculating expectations often requires evaluating integrals with respect to a Gaussian measure. Monte Carlo methods are widely used for this purpose due to their dimension-independent convergence rate. Quasi-Monte Carlo is the deterministic analogue of Monte Carlo and has the potential to substantially enhance the convergence rate. Importance sampling is a widely used variance reduction technique. However, research into the specific impact of importance sampling on the integrand, as well as the conditions for convergence, is relatively scarce. In this study, we combine the randomly shifted lattice rule with importance sampling. We prove that, for unbounded functions, randomly shifted lattice rules combined with a suitably chosen importance density can achieve convergence as quickly as O(N−1+ϵ), given N samples for arbitrary ϵ values under certain conditions. We also prove that the conditions of convergence for Laplace importance sampling are stricter than those for optimal drift importance sampling. Furthermore, using a generalized linear mixed model and Randleman–Bartter model, we provide the conditions under which functions utilizing Laplace importance sampling achieve convergence rates of nearly O(N−1+ϵ) for arbitrary ϵ values.

Analyticity of parametric elliptic eigenvalue problems and applications to quasi-Monte Carlo methods

In the present paper, we study the analyticity of the leftmost eigenvalue of the linear elliptic partial differential operators with random coefficient and analyse the convergence rate of the quasi-Monte Carlo method for approximation of the expectation of this quantity. The random coefficient is assumed to be represented by an affine expansion a 0 ( x ) + ∑ j ∈ N y j a j ( x ) , where elements of the parameter vector y = ( y j ) j ∈ N ∈ U ∞ are independent and identically uniformly distributed on U := [ − 1 2 , 1 2 ] . Under the assumption ‖ ∑ j ∈ N ρ j | a j | ‖ L ∞ ( D ) < ∞ with some positive sequence ( ρ j ) j ∈ N ∈ ℓ p ( N ) for p ∈ ( 0 , 1 ] we show that for any y ∈ U ∞ , the elliptic partial differential operator has a countably infinite number of eigenvalues ( λ j ( y ) ) j ∈ N which can be ordered non-decreasingly. Moreover, the spectral gap λ 2 ( y ) − λ 1 ( y ) is uniformly positive in U ∞ . From this, we prove the holomorphic extension property of λ 1 ( y ) to a complex domain in C ∞ and estimate partial derivatives of λ 1 ( y ) with respect to the parameter y by using Cauchy's formula for analytic functions. Based on these bounds we prove the dimension-independent convergence rate of the quasi-Monte Carlo method to approximate the expectation of λ 1 ( y ) .

Density Estimation in RKHS with Application to Korobov Spaces in High Dimensions

A kernel method for estimating a probability density function from an independent and identically distributed sample drawn from such density is presented. Our estimator is a linear combination of kernel functions, the coefficients of which are determined by a linear equation. An error analysis for the mean integrated squared error is established in a general reproducing kernel Hilbert space setting. The theory developed is then applied to estimate probability density functions belonging to weighted Korobov spaces, for which a dimension-independent convergence rate is established. Under a suitable smoothness assumption, our method attains a rate arbitrarily close to the optimal rate. Numerical results support our theory.

Dimension-free local convergence and perturbations for reflected Brownian motions

We describe and analyze a class of positive recurrent reflected Brownian motions (RBMs) in R+d for which local statistics converge to equilibrium at a rate independent of the dimension d. Under suitable assumptions on the reflection matrix, drift and diffusivity coefficients, dimension-independent stretched exponential convergence rates are obtained by estimating contractions in an underlying weighted distance between synchronously coupled RBMs. We also study the symmetric Atlas model as a first step in obtaining dimension-independent convergence rates for RBMs not satisfying the above assumptions. By analyzing a pathwise derivative process and connecting it to a random walk in a random environment, we obtain polynomial convergence rates for the gap process of the symmetric Atlas model started from appropriate perturbations of stationarity.

Convergence and finite sample approximations of entropic regularized Wasserstein distances in Gaussian and RKHS settings

This work studies the convergence and finite sample approximations of entropic regularized Wasserstein distances in the Hilbert space setting. Our first main result is that for Gaussian measures on an infinite-dimensional Hilbert space, convergence in the 2-Sinkhorn divergence is strictly weaker than convergence in the exact 2-Wasserstein distance. Specifically, a sequence of centered Gaussian measures converges in the 2-Sinkhorn divergence if the corresponding covariance operators converge in the Hilbert–Schmidt norm. This is in contrast to the previous known result that a sequence of centered Gaussian measures converges in the exact 2-Wasserstein distance if and only if the covariance operators converge in the trace class norm. In the reproducing kernel Hilbert space (RKHS) setting, the kernel Gaussian–Sinkhorn divergence, which is the Sinkhorn divergence between Gaussian measures defined on an RKHS, defines a semi-metric on the set of Borel probability measures on a Polish space, given a characteristic kernel on that space. With the Hilbert–Schmidt norm convergence, we obtain dimension-independent convergence rates for finite sample approximations of the kernel Gaussian–Sinkhorn divergence, of the same order as the Maximum Mean Discrepancy. These convergence rates apply in particular to Sinkhorn divergence between Gaussian measures on Euclidean and infinite-dimensional Hilbert spaces. The sample complexity for the 2-Wasserstein distance between Gaussian measures on Euclidean space, while dimension-dependent, is exponentially faster than the worst case scenario in the literature.

Continuum limit and preconditioned Langevin sampling of the path integral molecular dynamics

We investigate the continuum limit that the number of beads goes to infinity in the ring polymer representation of thermal averages. Studying the continuum limit of the trajectory sampling equation sheds light on possible preconditioning techniques for sampling ring polymer configurations with large number of beads. We propose two preconditioned Langevin sampling dynamics, which are shown to have improved stability and sampling accuracy. We present a careful mode analysis of the preconditioned dynamics and show their connections to the normal mode, the staging coordinate and the Matsubara mode representation for ring polymers. In the case where the potential is quadratic, we show that the continuum limit of the preconditioned mass modified Langevin dynamics converges to its equilibrium exponentially fast, which suggests that the finite dimensional counterpart has a dimension-independent convergence rate. In addition, the preconditioning techniques can be naturally applied to the multi-level quantum systems in the nonadiabatic regime, which are compatible with various numerical approaches.

Convergence rates of high dimensional Smolyak quadrature

We analyse convergence rates of Smolyak integration for parametric maps u: U → X taking values in a Banach space X, defined on the parameter domain U = [−1,1]N. For parametric maps which are sparse, as quantified by summability of their Taylor polynomial chaos coefficients, dimension-independent convergence rates superior to N-term approximation rates under the same sparsity are achievable. We propose a concrete Smolyak algorithm to a priori identify integrand-adapted sets of active multiindices (and thereby unisolvent sparse grids of quadrature points) via upper bounds for the integrands’ Taylor gpc coefficients. For so-called “(b,ε)-holomorphic” integrands u with b∈lp(∕) for some p ∈ (0, 1), we prove the dimension-independent convergence rate 2/p − 1 in terms of the number of quadrature points. The proposed Smolyak algorithm is proved to yield (essentially) the same rate in terms of the total computational cost for both nested and non-nested univariate quadrature points. Numerical experiments and a mathematical sparsity analysis accounting for cancellations in quadratures and in the combination formula demonstrate that the asymptotic rate 2/p − 1 is realized computationally for a moderate number of quadrature points under certain circumstances. By a refined analysis of model integrand classes we show that a generally large preasymptotic range otherwise precludes reaching the asymptotic rate 2/p − 1 for practically relevant numbers of quadrature points.

On the Banach Spaces Associated with Multi-Layer ReLU Networks: Function Representation, Approximation Theory and Gradient Descent Dynamics

We develop Banach spaces for ReLU neural networks of finite depth $L$ and infinite width. The spaces contain all finite fully connected $L$-layer networks and their $L^2$-limiting objects under bounds on the natural path-norm. Under this norm, the unit ball in the space for $L$-layer networks has low Rademacher complexity and thus favorable generalization properties. Functions in these spaces can be approximated by multi-layer neural networks with dimension-independent convergence rates. The key to this work is a new way of representing functions in some form of expectations, motivated by multi-layer neural networks. This representation allows us to define a new class of continuous models for machine learning. We show that the gradient flow defined this way is the natural continuous analog of the gradient descent dynamics for the associated multi-layer neural networks. We show that the path-norm increases at most polynomially under this continuous gradient flow dynamics.

Multilevel approximation of parametric and stochastic PDES

We analyze the complexity of the sparse-grid interpolation and sparse-grid quadrature of countably-parametric functions which take values in separable Banach spaces with unconditional bases. Assuming a suitably quantified holomorphic dependence on the parameters, we establish dimension-independent convergence rate bounds for sparse-grid approximation schemes. Analogous results are shown in the case that the parametric families are obtained as approximate solutions of corresponding parametric-holomorphic, nonlinear operator equations as considered in [A. Cohen and A. Chkifa and Ch. Schwab: Breaking the curse of dimensionality in sparse polynomial approximation of parametric PDEs, J. Math. Pures Appl. 103 (2015) 400–428], for example by means of stable, finite-dimensional approximations. We discuss in detail nonlinear Petrov–Galerkin projections. Error and convergence rate bounds for constructive and explicit multilevel, sparse tensor approximation schemes combining sparse-grid interpolation in the parameter space and general, multilevel discretization schemes in the physical domain are proved. The present results unify and generalize earlier works in terms of the admissible multilevel approximations in the physical domain (comprising general stable Petrov–Galerkin and discrete Petrov–Galerkin schemes, collocation and stable domain approximations) and in terms of the admissible operator equations (comprising smooth, nonlinear locally well-posed operator equations). Additionally, a novel computational strategy to localize sequences of nested index sets for the anisotropic Smolyak interpolation in parameter space is developed which realizes best [Formula: see text]-term benchmark convergence rates. We also consider Smolyak-type quadratures in this general setting, for which we establish improved convergence rates based on cancellations in the integrands’ gpc expansions by symmetries of quadratures and the probability measure [J.Z̃ech and Ch.S̃chwab: Convergence rates of high dimensional Smolyak quadrature, Report 2017-27, SAM ETH Zürich (2017)]. Several examples illustrating the abstract theory include domain uncertainty quantification (UQ) for general, linear, second-order, elliptic advection–reaction–diffusion equations on polygonal domains, where optimal convergence rates of FEM are known to require local mesh refinement near corners. Further applications of the presently developed theory comprise evaluations of posterior expectations in Bayesian inverse problems.

Higher order Quasi-Monte Carlo integration for Bayesian PDE Inversion

We analyze combined Quasi-Monte Carlo quadrature and Finite Element approximations in Bayesian estimation of solutions to countably-parametric operator equations with holomorphic dependence on the parameters as considered in Schillings and Schwab (2014). Such problems arise in numerical uncertainty quantification and in Bayesian inversion of operator equations with distributed uncertain inputs, such as uncertain coefficients, uncertain domains or uncertain source terms and boundary data. We show that the parametric Bayesian posterior densities belong to a class of weighted Bochner spaces of functions of countably many variables, with a particular structure of the QMC quadrature weights: up to a (problem-dependent, and possibly large) finite dimension S product weights can be used, and beyond this dimension, weighted spaces with so-called SPOD weights, recently introduced in Dick et al. (2014), are used to describe the solution regularity. We establish error bounds for higher order Quasi-Monte Carlo quadrature for the Bayesian estimation based on Dick et al. (2016). It implies, in particular, regularity of the parametric solution and of the countably-parametric Bayesian posterior density in SPOD (“Smoothness driven, Product and Order Dependent”) weighted spaces of integrand functions. This, in turn, implies that the Quasi-Monte Carlo quadrature methods in Dick et al. (2014) are applicable to these problem classes, with dimension-independent convergence rates O(N−1∕p) of N-point HoQMC approximated Bayesian estimates, where 0<p<1 depends only on the sparsity class of the uncertain input in the Bayesian estimation. Fast component-by-component (CBC for short) construction Gantner and Schwab (2016) allow efficient deterministic Bayesian estimation with up to 104 parameters.

Sparse quadrature for high-dimensional integration with Gaussian measure

In this work we analyze the dimension-independent convergence property of an abstract sparse quadrature scheme for numerical integration of functions of high-dimensional parameters with Gaussian measure. Under certain assumptions on the exactness and boundedness of univariate quadrature rules as well as on the regularity assumptions on the parametric functions with respect to the parameters, we prove that the convergence of the sparse quadrature error is independent of the number of the parameter dimensions. Moreover, we propose both an a priori and an a posteriori schemes for the construction of a practical sparse quadrature rule and perform numerical experiments to demonstrate their dimension-independent convergence rates.

Quasi--Monte Carlo Integration for Affine-Parametric, Elliptic PDEs: Local Supports and Product Weights

We analyze convergence rates of first-order quasi--Monte Carlo (QMC) integration with randomly shifted lattice rules and for higher-order, interlaced polynomial lattice rules for a class of countably parametric integrands that result from linear functionals of solutions of linear, elliptic diffusion equations with affine-parametric, uncertain coefficient function $a(x,{y}) = \bar{a}(x) + \sum_{j\geq 1} y_j \psi_j(x)$ in a bounded domain $D\subset \mathbb{R}^d$. Extending the result in [F. Y. Kuo, C. Schwab, and I. H. Sloan, SIAM J. Numer. Anal., 50 (2012), pp. 3351--3374], where $\psi_j$ was assumed to have global support in the domain $D$, we assume in the present paper that ${supp}(\psi_j)$ is localized in $D$ and that we have control on the overlaps of these supports. Under these conditions we prove dimension-independent convergence rates in [1/2,1) of randomly shifted lattice rules with product weights and corresponding higher-order convergence rates by higher-order, interlaced polynomial lattice rules with product weights. The product structure of the QMC weights facilitates work bounds for the fast, component-by-component constructions of [D. Nuyens and R. Cools, Math. Comp., 75 (2006), pp. 903--920] which scale linearly with respect to the parameter dimension $s$. The dimension-independent convergence rates are only limited by the degree of digit interlacing used in the construction of the higher-order QMC quadrature rule and, for locally supported coefficient functions, by the summability of the locally supported coefficient sequence in the affine-parametric coefficient.

Convergence of Sparse Collocation for Functions of Countably Many Gaussian Random Variables (with Application to Elliptic PDEs)

We give a convergence proof for the approximation by sparse collocation of Hilbert-space-valued functions depending on countably many Gaussian random variables. Such functions appear as solutions of elliptic PDEs with lognormal diffusion coefficients. We outline a general $L^2$-convergence theory based on previous work by Bachmayr et al. [ESAIM Math. Model. Numer. Anal., 51 (2017), pp. 341--363] and Chen [ESAIM Math. Model. Numer. Anal., in press, 2018, https://doi.org/10.1051/m2an/2018012] and establish an algebraic convergence rate for sufficiently smooth functions assuming a mild growth bound for the univariate hierarchical surpluses of the interpolation scheme applied to Hermite polynomials. We specifically verify for Gauss--Hermite nodes that this assumption holds and also show algebraic convergence with respect to the resulting number of sparse grid points for this case. Numerical experiments illustrate the dimension-independent convergence rate.

Electromagnetic wave scattering by random surfaces: Shape holomorphy

For time-harmonic electromagnetic waves scattered by either perfectly conducting or dielectric bounded obstacles, we show that the fields depend holomorphically on the shape of the scatterer. In the presence of random geometrical perturbations, our results imply strong measurability of the fields, in weighted spaces in the exterior of the scatterer. These findings are key to prove dimension-independent convergence rates of sparse approximation techniques of polynomial chaos type for forward and inverse computational uncertainty quantification. Also, our shape-holomorphy results imply parsimonious approximate representations of the corresponding parametric solution families, which are produced, for example, by greedy strategies such as model order reduction or reduced basis approximations. Finally, the presently proved shape holomorphy results imply convergence of shape Taylor expansions of far-field patterns for fixed amplitude domain perturbations in a vicinity of the nominal domain, thereby extending the widely used asymptotic linearizations employed in first-order, second moment domain uncertainty quantification.

Hessian-based adaptive sparse quadrature for infinite-dimensional Bayesian inverse problems

In this work we propose and analyze a Hessian-based adaptive sparse quadrature to compute infinite-dimensional integrals with respect to the posterior distribution in the context of Bayesian inverse problems with Gaussian prior. Due to the concentration of the posterior distribution in the domain of the prior distribution, a prior-based parametrization and sparse quadrature may fail to capture the posterior distribution and lead to erroneous evaluation results. By using a parametrization based on the Hessian of the negative log-posterior, the adaptive sparse quadrature can effectively allocate the quadrature points according to the posterior distribution. A dimension-independent convergence rate of the proposed method is established under certain assumptions on the Gaussian prior and the integrands. Dimension-independent and faster convergence than O(N−1∕2) is demonstrated for a linear as well as a nonlinear inverse problem whose posterior distribution can be effectively approximated by a Gaussian distribution at the MAP point.

Sparse-grid, reduced-basis Bayesian inversion: Nonaffine-parametric nonlinear equations

We extend the reduced basis (RB) accelerated Bayesian inversion methods for affine-parametric, linear operator equations which are considered in 16,17 to non-affine, nonlinear parametric operator equations. We generalize the analysis of sparsity of parametric forward solution maps in 20 and of Bayesian inversion in 48,49 to the fully discrete setting, including Petrov-Galerkin high-fidelity (HiFi) discretization of the forward maps. We develop adaptive, stochastic collocation based reduction methods for the efficient computation of reduced bases on the parametric solution manifold. The nonaffinity and nonlinearity with respect to (w.r.t.) the distributed, uncertain parameters and the unknown solution is collocated; specifically, by the so-called Empirical Interpolation Method (EIM). For the corresponding Bayesian inversion problems, computational efficiency is enhanced in two ways: first, expectations w.r.t. the posterior are computed by adaptive quadratures with dimension-independent convergence rates proposed in 49; the present work generalizes 49 to account for the impact of the PG discretization in the forward maps on the convergence rates of the Quantities of Interest (QoI for short). Second, we propose to perform the Bayesian estimation only w.r.t. a parsimonious, RB approximation of the posterior density. Based on the approximation results in 49, the infinite-dimensional parametric, deterministic forward map and operator admit N-term RB and EIM approximations which converge at rates which depend only on the sparsity of the parametric forward map. In several numerical experiments, the proposed algorithms exhibit dimension-independent convergence rates which equal, at least, the currently known rate estimates for N-term approximation. We propose to accelerate Bayesian estimation by first offline construction of reduced basis surrogates of the Bayesian posterior density. The parsimonious surrogates can then be employed for online data assimilation and for Bayesian estimation. They also open a perspective for optimal experimental design.

Higher Order QMC Petrov--Galerkin Discretization for Affine Parametric Operator Equations with Random Field Inputs

We construct quasi--Monte Carlo methods to approximate the expected values of linear functionals of Petrov--Galerkin discretizations of parametric operator equations which depend on a possibly infinite sequence of parameters. Such problems arise in the numerical solution of differential and integral equations with random field inputs. We analyze the regularity of the solutions with respect to the parameters in terms of the rate of decay of the fluctuations of the input field. If $p\in (0,1]$ denotes the “summability exponent” corresponding to the fluctuations in affine-parametric families of operators, then we prove that deterministic “interlaced polynomial lattice rules” of order $\alpha = \lfloor 1/p \rfloor+1$ in $s$ dimensions with $N$ points can be constructed using a fast component-by-component algorithm, in $\mathcal{O}(\alpha\,s\, N\log N + \alpha^2\,s^2 N)$ operations, to achieve a convergence rate of $\mathcal{O}(N^{-1/p})$, with the implied constant independent of $s$. This dimension-independent convergence rate is superior to the rate $\mathcal{O}(N^{-1/p+1/2})$ for $2/3\leq p\leq 1$, which was recently established for randomly shifted lattice rules under comparable assumptions. In our analysis we use a nonstandard Banach space setting and introduce “smoothness-driven product and order dependent” weights for which we develop a new fast component-by-component construction.

Convergence of a two-level ideal algorithm for a parametric shape inverse model problem

Similar to the discretization of ordinary or partial differential equations, the numerical approximation of the solution of an optimization problem is possibly subject to numerical stiffness. In the framework of parametric shape optimization, hierarchical representations of the shape can be used for preconditioning, following the idea of Multigrid (MG) methods. In this article, by analogy with the Poisson equation, which is the typical example for linear MG methods, we address a parametric shape inverse problem. We describe the ideal cycle of a two-level algorithm adapted to shape optimization problems that require appropriate transfer operators. With the help of a symbolic calculus software we show that the efficiency of an optimization MG-like strategy is ensured by a small dimension-independent convergence rate. Numerical examples are worked out and corroborate the theoretical results. Applications to antenna design are realized. Finally, some connections with the direct and inverse Broyden–Fletcher–Goldfarb–Shanno preconditioning methods are shown.

On Dimension-independent Rates of Convergence for Function Approximation with Gaussian Kernels

This article studies the problem of approximating functions belonging to a Hilbert space $H_d$ with an isotropic or anisotropic Gaussian reproducing kernel, $$ K_d(\bx,\bt) = \exp\left(-\sum_{\ell=1}^d\gamma_\ell^2(x_\ell-t_\ell)^2\right) \ \ \ \mbox{for all}\ \ \bx,\bt\in\reals^d. $$ The isotropic case corresponds to using the same shape parameters for all coordinates, namely $\gamma_\ell=\gamma>0$ for all $\ell$, whereas the anisotropic case corresponds to varying shape parameters $\gamma_\ell$. We are especially interested in moderate to large $d$.

Sparse Tensor Product Wavelet Approximation of Singular Functions

On product domains, sparse-grid approximation yields optimal, dimension-independent convergence rates when the function that is approximated has $L^2$-bounded mixed derivatives of a sufficiently high order. We show that the solution of Poisson's equation on the n-dimensional hypercube with Dirichlet boundary conditions and smooth right-hand side generally does not satisfy this condition. As suggested by P.-A. Nitsche in [Constr. Approx., 21 (2005), pp. 63–81], the regularity conditions can be relaxed to corresponding ones in weighted $L^2$ spaces when the sparse-grid approach is combined with local refinement of the set of one-dimensional wavelet indices towards the end points. In this paper, we prove that for general smooth right-hand sides, the solution of Poisson's problem satisfies these relaxed regularity conditions in any space dimension. Furthermore, since we remove log-factors from the energy-error estimates from Nitsche's work, we show that in any space dimension, locally refined sparse-grid approximation yields the optimal, dimension-independent convergence rate.