Infinite-dimensional Parameter Space Research Articles

Adaptive reduced basis trust region methods for parameter identification problems

In this contribution, we are concerned with model order reduction in the context of iterative regularization methods for the solution of inverse problems arising from parameter identification in elliptic partial differential equations. Such methods typically require a large number of forward solutions, which makes the use of the reduced basis method attractive to reduce computational complexity. However, the considered inverse problems are typically ill-posed due to their infinite-dimensional parameter space. Moreover, the infinite-dimensional parameter space makes it impossible to build and certify classical reduced-order models efficiently in a so-called “offline phase”. We thus propose a new algorithm that adaptively builds a reduced parameter space in the online phase. The enrichment of the reduced parameter space is naturally inherited from the Tikhonov regularization within an iteratively regularized Gauß-Newton method. Finally, the adaptive parameter space reduction is combined with a certified reduced basis state space reduction within an adaptive error-aware trust region framework. Numerical experiments are presented to show the efficiency of the combined parameter and state space reduction for inverse parameter identification problems with distributed reaction or diffusion coefficients.

A DIMENSION-ADAPTIVE COMBINATION TECHNIQUE FOR UNCERTAINTY QUANTIFICATION

We present an adaptive algorithm for the computation of quantities of interest involving the solution of a stochastic elliptic partial differential equation, where the diffusion coefficient is parametrized by means of a Karhunen-Loève expansion. The approximation of the equivalent parametric problem requires a restriction of the countably infinite-dimensional parameter space to a finite-dimensional parameter set, a spatial discretization, and an approximation in the parametric variables. We consider a sparse grid approach between these approximation directions in order to reduce the computational effort and propose a dimension-adaptive combination technique. In addition, a sparse grid quadrature for the high-dimensional parametric approximation is employed and simultaneously balanced with the spatial and stochastic approximation. Our adaptive algorithm constructs a sparse grid approximation based on the benefit-cost ratio such that the regularity and thus the decay of the Karhunen-Loève coefficients is not required beforehand. The decay is detected and exploited as the algorithm adjusts to the anisotropy in the parametric variables. We include numerical examples for the Darcy problem with a lognormal permeability field, which illustrate a good performance of the algorithm. For sufficiently smooth random fields, we essentially recover the spatial order of convergence as asymptotic convergence rate with respect to the computational cost.

Two-layer neural network on infinite-dimensional data: global optimization guarantee in the mean-field regime *

The analysis of neural network optimization in the mean-field regime is important as the setting allows for feature learning. The existing theory has been developed mainly for neural networks in finite dimensions, i.e. each neuron has a finite-dimensional parameter. However, the setting of infinite-dimensional input naturally arises in machine learning problems such as nonparametric functional data analysis and graph classification. In this paper, we develop a new mean-field analysis of a two-layer neural network in an infinite-dimensional parameter space. We first give a generalization error bound, which shows that the regularized empirical risk minimizer properly generalizes when the data size is sufficiently large, despite the neurons being infinite-dimensional. Next, we present two gradient-based optimization algorithms for infinite-dimensional mean-field networks, by extending the recently developed particle optimization framework to the infinite-dimensional setting. We show that the proposed algorithms converge to the (regularized) global optimal solution, and moreover, their rates of convergence are of polynomial order in the online setting and exponential order in the finite sample setting, respectively. To the best of our knowledge, this is the first quantitative global optimization guarantee of a neural network on infinite-dimensional input and in the presence of feature learning.

Sieve estimation of a class of partially linear transformation models with interval-censored competing risks data.

In this paper, we consider a class of partially linear transformation models with interval-censored competing risks data. Under a semiparametric generalized odds rate specification for the cause-specific cumulative incidence function, we obtain optimal estimators of the large number of parametric and nonparametric model components via maximizing the likelihood function over a joint B-spline and Bernstein polynomial spanned sieve space. Our specification considers a relatively simpler finite-dimensional parameter space, approximating the infinite-dimensional parameter space as n → ∞, thereby allowing us to study the almost sure consistency, and rate of convergence for all parameters, and the asymptotic distributions and efficiency of the finite-dimensional components. We study the finite sample performance of our method through simulation studies under a variety of scenarios. Furthermore, we illustrate our methodology via application to a dataset on HIV-infected individuals from sub-Saharan Africa.

Infinite hidden Markov model for short-term solar irradiance forecasting

Hidden state models are among the most widely used and efficient schemes for solar irradiance modeling in general and forecasting in particular. However, the complexity of such models – in terms of the number of states – is usually needed to be specified a priori. For solar irradiance data this assumption is very difficult to justify.In this paper, an infinite hidden Markov model (InfHMM) is introduced for short-term probabilistic forecasting of solar irradiance, where the assumption of fixed number of states a priori is relaxed and model complexity is determined during the model training. InfHMM is a non-parametric Bayesian model (NPB) indexed with an infinite dimensional parameter space which allows the automatic adaptation of the model to the “correct” complexity. This facilitates the automatic adaptation of the model to all weather conditions and locations. Posterior inference for InfHMM is performed using the Markov chain Monte Carlo algorithm, namely the beam sampler.Data from 13 different sources are used to validate the proposed model and subsequently it is compared to two well-established models in the literature: Markov-chain mixture distribution (MCM) and complete-history persistence ensemble (CH-PeEn) models. Important results are found, that cannot be derived from the existing finite models, such as the variation of the number of states within and across sites. The comparison of the models shows that the InfHMM is more consistent in term of the forecasting horizon.For reproducibility of the methodology presented in this paper, we have provided an R script for the InfHMM as supplementary material.

Strongly Nonlinear Diffusion in Turbulent Environment: A Problem with Infinitely Many Couplings

The field theoretic renormalization group is applied to the strongly nonlinear stochastic advection-diffusion equation. The turbulent advection is modelled by the Kazantsev–Kraichnan “rapid-change” ensemble. As a requirement of the renormalizability, the model necessarily involves infinite number of coupling constants (“charges”). The one-loop counterterm is calculated explicitly. The corresponding renormalization group equation demonstrates existence of a pair of two-dimensional surfaces of fixed points in the infinite-dimensional parameter space. If the surfaces contain infrared attractive regions, the problem allows for the large-scale, long-time scaling behaviour. For the first surface (advection is irrelevant), the critical dimensions of the scalar field Δθ, the response field Δθ′ and the frequency Δω are nonuniversal (through the dependence on the effective couplings) but satisfy certain exact identities. For the second surface (advection is relevant), the dimensions are universal and they are found exactly.

Nonlinear random vibration analysis: A Bayesian nonparametric approach

Random vibration analysis aims to estimate the response statistics of dynamical systems subject to stochastic excitations. Stochastic differential equations (SDEs) that govern the response of general nonlinear systems are often complicated, and their analytical solutions are scarce. Thus, a range of approximate methods and simulation techniques have been developed. This paper develops a hybrid approach that approximates the governing SDE of nonlinear systems using a small number of response simulations and information available a priori. The main idea is to identify a set of surrogate linear systems such that their response probability distributions collectively estimate the response probability distribution of the original nonlinear system. To identify the surrogate linear systems, the proposed method integrates the simulated responses of the original nonlinear system with information available a priori about the number and parameters of the surrogate linear systems. There will be epistemic uncertainty in the number and parameters of the surrogate linear systems because of the limited data. This paper proposes a Bayesian nonparametric approach, called a Dirichlet Process Mixture Model, to capture these uncertainties. The Dirichlet process models the uncertainty over an infinite-dimensional parameter space, representing an infinite number of potential surrogate linear systems. Specifically, the proposed method allows the number of surrogate linear systems to grow indefinitely as the nonlinear system observed dynamic unveil new patterns. The quantified uncertainty in the estimates of the unknown model parameters propagates into the response probability distribution. The paper then shows that, under some mild conditions, the estimated probability distribution approaches, as close as desired, to the original nonlinear system’s response probability distribution. As a measure of model accuracy, the paper provides the convergence rate of the response probability distribution. Because the posterior distribution of the unknown model parameters is often not analytically tractable, a Gibbs sampling algorithm is presented to draw samples from the posterior distribution. Variational Bayesian inference is also introduced to derive an approximate closed-form expression for the posterior distribution. The paper illustrates the proposed method through the random vibration analysis of a nonlinear elastic and a nonlinear hysteretic system.

Shrinkage Priors for Nonparametric Bayesian Prediction of Nonhomogeneous Poisson Processes

We consider nonparametric Bayesian estimation and prediction for nonhomogeneous Poisson process models with unknown intensity functions. We propose a class of improper priors for intensity functions. Nonparametric Bayesian inference with kernel mixture based on the class improper priors is shown to be useful, although improper priors have not been widely used for nonparametric Bayes problems. Several theorems corresponding to those for finite-dimensional independent Poisson models hold for nonhomogeneous Poisson process models with infinite-dimensional parameter spaces. Bayesian estimation and prediction based on the improper priors are shown to be admissible under the Kullback-Leibler loss. Numerical methods for Bayesian inference based on the priors are investigated.

A fast Chebyshev method for simulating flexible-wing propulsion

We develop a highly efficient numerical method to simulate small-amplitude flapping propulsion by a flexible wing in a nearly inviscid fluid. We allow the wing's elastic modulus and mass density to vary arbitrarily, with an eye towards optimizing these distributions for propulsive performance. The method to determine the wing kinematics is based on Chebyshev collocation of the 1D beam equation as coupled to the surrounding 2D fluid flow. Through small-amplitude analysis of the Euler equations (with trailing-edge vortex shedding), the complete hydrodynamics can be represented by a nonlocal operator that acts on the 1D wing kinematics. A class of semi-analytical solutions permits fast evaluation of this operator with O(Nlog⁡N) operations, where N is the number of collocation points on the wing. This is in contrast to the minimum O(N2) cost of a direct 2D fluid solver. The coupled wing–fluid problem is thus recast as a PDE with nonlocal operator, which we solve using a preconditioned iterative method. These techniques yield a solver of near-optimal complexity, O(Nlog⁡N), allowing one to rapidly search the infinite-dimensional parameter space of all possible material distributions and even perform optimization over this space.

Well-posed Bayesian inverse problems and heavy-tailed stable quasi-Banach space priors

This article extends the framework of Bayesian inverse problems in infinite-dimensional parameter spaces, as advocated by Stuart (Acta Numer. 19:451--559, 2010) and others, to the case of a heavy-tailed prior measure in the family of stable distributions, such as an infinite-dimensional Cauchy distribution, for which polynomial moments are infinite or undefined. It is shown that analogues of the Karhunen--Lo\`eve expansion for square-integrable random variables can be used to sample such measures on quasi-Banach spaces. Furthermore, under weaker regularity assumptions than those used to date, the Bayesian posterior measure is shown to depend Lipschitz continuously in the Hellinger metric upon perturbations of the misfit function and observed data.

FEM-based discretization-invariant MCMC methods for PDE-constrained Bayesian inverse problems

We present a systematic construction of FEM-based dimension-independent (discretization-invariant) Markov chain Monte Carlo (MCMC) approaches to explore PDE-constrained Bayesian inverse problems in infinite dimensional parameter spaces. In particular, we consider two frameworks to achieve this goal: Metropolize-then-discretize and discretize-then-Metropolize. The former refers to the method of discretizing function-space MCMC methods. The latter, on the other hand, first discretizes the Bayesian inverse problem and then proposes MCMC methods for the resulting discretized posterior probability density. In general, these two frameworks do not commute, that is, the resulting finite dimensional MCMC algorithms are not identical. The discretization step of the former may not be trivial since it involves both numerical analysis and probability theory, while the latter, perhaps ``easier'', may not be discretization-invariant using traditional approaches. This paper constructively develops finite element (FEM) discretization schemes for both frameworks and shows that both commutativity and discretization-invariance are attained. In particular, it shows how to construct discretize-then-Metropolize approaches for both Metropolis-adjusted Langevin algorithm and the hybrid Monte Carlo method that commute with their Metropolize-then-discretize counterparts. The key that enables this achievement is a proper FEM discretization of the prior, the likelihood, and the Bayes' formula, together with a correct definition of quantities such as the gradient and the covariance matrix in discretized finite dimensional parameter spaces. The implication is that practitioners can take advantage of the developments in this paper to straightforwardly construct discretization-invariant discretize-then-Metropolize MCMC for large-scale inverse problems. Numerical results for one- and two-dimensional elliptic inverse problems with up to $17899$ parameters are presented to support the proposed developments.

Solving large-scale PDE-constrained Bayesian inverse problems with Riemann manifold Hamiltonian Monte Carlo

We consider the Riemann manifold Hamiltonian Monte Carlo (RMHMC) method for solving statistical inverse problems governed by partial differential equations (PDEs). The Bayesian framework is employed to cast the inverse problem into the task of statistical inference whose solution is the posterior distribution in infinite dimensional parameter space conditional upon observation data and Gaussian prior measure. We discretize both the likelihood and the prior using the H1-conforming finite element method together with a matrix transfer technique. The power of the RMHMC method is that it exploits the geometric structure induced by the PDE constraints of the underlying inverse problem. Consequently, each RMHMC posterior sample is almost uncorrelated/independent from the others providing statistically efficient Markov chain simulation. However this statistical efficiency comes at a computational cost. This motivates us to consider computationally more efficient strategies for RMHMC. At the heart of our construction is the fact that for Gaussian error structures the Fisher information matrix coincides with the Gauss–Newton Hessian. We exploit this fact in considering a computationally simplified RMHMC method combining state-of-the-art adjoint techniques and the superiority of the RMHMC method. Specifically, we first form the Gauss–Newton Hessian at the maximum a posteriori point and then use it as a fixed constant metric tensor throughout RMHMC simulation. This eliminates the need for the computationally costly differential geometric Christoffel symbols, which in turn greatly reduces computational effort at a corresponding loss of sampling efficiency. We further reduce the cost of forming the Fisher information matrix by using a low rank approximation via a randomized singular value decomposition technique. This is efficient since a small number of Hessian-vector products are required. The Hessian-vector product in turn requires only two extra PDE solves using the adjoint technique. Various numerical results up to 1025 parameters are presented to demonstrate the ability of the RMHMC method in exploring the geometric structure of the problem to propose (almost) uncorrelated/independent samples that are far away from each other, and yet the acceptance rate is almost unity. The results also suggest that for the PDE models considered the proposed fixed metric RMHMC can attain almost as high a quality performance as the original RMHMC, i.e. generating (almost) uncorrelated/independent samples, while being two orders of magnitude less computationally expensive.

BEST N-TERM GPC APPROXIMATIONS FOR A CLASS OF STOCHASTIC LINEAR ELASTICITY EQUATIONS

We consider a class of stochastic linear elasticity problems whose elastic moduli depend linearly on a countable set of random variables. The stochastic equation is studied via a deterministic parametric problem on an infinite-dimensional parameter space. We first study the best N-term approximation of the generalized polynomial chaos (gpc) expansion of the solution to the displacement formula by considering a Galerkin projection onto the space obtained by truncating the gpc expansion. We provide sufficient conditions on the coefficients of the elastic moduli's expansion so that a rate of convergence for this approximation holds. We then consider two classes of stochastic and parametric mixed elasticity problems. The first one is the Hellinger–Reissner formula for approximating directly the gpc expansion of the stress. For isotropic problems, the multiplying constant of the best N-term convergence rate for the displacement formula grows with the ratio of the Lamé constants. We thus consider stochastic and parametric mixed problems for nearly incompressible isotropic materials whose best N-term approximation rate is uniform with respect to the ratio of the Lamé constants.

Adaptive stochastic Galerkin FEM

A framework for residual-based a posteriori error estimation and adaptive mesh refinement and polynomial chaos expansion for general second order linear elliptic PDEs with random coefficients is presented. A parametric, deterministic elliptic boundary value problem on an infinite-dimensional parameter space is discretized by means of a Galerkin projection onto finite generalized polynomial chaos (gpc) expansions, and by discretizing each gpc coefficient by a FEM in the physical domain.An anisotropic residual-based a posteriori error estimator is developed. It contains bounds for both contributions to the overall error: the error due to gpc discretization and the error due to Finite Element discretization of the gpc coefficients in the expansion. The reliability of the residual estimator is established.Based on the explicit form of the residual estimator, an adaptive refinement strategy is presented which allows to steer the polynomial degree adaptation and the dimension adaptation in the stochastic Galerkin discretization, and, embedded in the gpc adaptation loop, also the Finite Element mesh refinement of the gpc coefficients in the physical domain. Asynchronous mesh adaptation for different gpc coefficients is permitted, subject to a minimal compatibility requirement on meshes used for different gpc coefficients.Details on the implementation with the open-source software framework ALEA are presented; it is generic, and is based on available stiffness and mass matrices of a FEM for the deterministic, nonparametric nominal problem evaluated in the FEniCS environment.Preconditioning of the resulting matrix equation and iterative solution are discussed. Numerical experiments in two spatial dimensions for membrane and plane stress boundary value problems on polygons are presented. They indicate substantial savings in total computational complexity due to FE mesh coarsening in high gpc coefficients.

Sparse Tensor Galerkin Discretization of Parametric and Random Parabolic PDEs---Analytic Regularity and Generalized Polynomial Chaos Approximation

For initial boundary value problems of linear parabolic partial differential equations with random coefficients, we show analyticity of the solution with respect to the parameters and give an a priori error analysis for $N$-term generalized polynomial chaos approximations in a scale of Bochner spaces. The problem is reduced to a parametric family of deterministic initial boundary value problems on an infinite dimensional parameter space by Galerkin projection onto finitely supported polynomial systems in the parameter space. Uniform stability with respect to the support of the resulting coupled parabolic systems is established. Analyticity of the solution with respect to the countably many parameters is established, and a regularity result of the parametric solution is proved for both compatible as well as incompatible initial data and source terms. The present results imply convergence rates and stability of sparse, adaptive space-time tensor product Galerkin discretizations of these infinite dimensional, parametric problems in the parameter space recently proposed in [C. Schwab and C. J. Gittelson, Acta Numer., 20 (2011), pp. 291--467; C. J. Gittelson, Adaptive Galerkin Methods for Parametric and Stochastic Operator Equations, Ph.D. thesis, ETH Zürich, 2011].

REGULARITY AND GENERALIZED POLYNOMIAL CHAOS APPROXIMATION OF PARAMETRIC AND RANDOM SECOND-ORDER HYPERBOLIC PARTIAL DIFFERENTIAL EQUATIONS

Initial boundary value problems of linear second-order hyperbolic partial differential equations whose coefficients depend on countably many random parameters are reduced to a parametric family of deterministic initial boundary value problems on an infinite dimensional parameter space. This parametric family is approximated by Galerkin projection onto finitely supported polynomial systems in the parameter space. We establish uniform stability with respect to the support of the resulting coupled hyperbolic systems, and provide sufficient smoothness and compatibility conditions on the data for the solution to exhibit analytic, respectively, Gevrey regularity with respect to the countably many parameters. Sufficient conditions for the p-summability of the generalized polynomial chaos expansion of the parametric solution in terms of the countably many input parameters are obtained and rates of convergence of best N-term polynomial chaos type approximations of the parametric solution are given. In addition, regularity both in space and time for the parametric family of solutions is proved for data satisfying certain compatibility conditions. The results allow obtaining convergence rates and stability of sparse space-time tensor product Galerkin discretizations in the parameter space.

Estimation of Nonparametric Conditional Moment Models With Possibly Nonsmooth Generalized Residuals

This paper studies nonparametric estimation of conditional moment restrictions in which the generalized residual functions can be nonsmooth in the unknown functions of endogenous variables. This is a nonparametric nonlinear instrumental variables (IV) problem. We propose a class of penalized sieve minimum distance (PSMD) estimators, which are minimizers of a penalized empirical minimum distance criterion over a collection of sieve spaces that are dense in the infinite-dimensional function parameter space. Some of the PSMD procedures use slowly growing finite-dimensional sieves with flexible penalties or without any penalty; others use large dimensional sieves with lower semicompact and/or convex penalties. We establish their consistency and the convergence rates in Banach space norms (such as a sup-norm or a root mean squared norm), allowing for possibly noncompact infinite-dimensional parameter spaces. For both mildly and severely ill-posed nonlinear inverse problems, our convergence rates in Hilbert space norms (such as a root mean squared norm) achieve the known minimax optimal rate for the nonparametric mean IV regression. We illustrate the theory with a nonparametric additive quantile IV regression. We present a simulation study and an empirical application of estimating nonparametric quantile IV Engel curves.

Empirical Bayes predictive densities for high-dimensional normal models

This paper addresses the problem of estimating the density of a future outcome from a multivariate normal model. We propose a class of empirical Bayes predictive densities and evaluate their performances under the Kullback–Leibler (KL) divergence. We show that these empirical Bayes predictive densities dominate the Bayesian predictive density under the uniform prior and thus are minimax under some general conditions. We also establish the asymptotic optimality of these empirical Bayes predictive densities in infinite-dimensional parameter spaces through an oracle inequality.

Sparse tensor discretizations of high-dimensional parametric and stochastic PDEs

Partial differential equations (PDEs) with random input data, such as random loadings and coefficients, are reformulated as parametric, deterministic PDEs on parameter spaces of high, possibly infinite dimension. Tensorized operator equations for spatial and temporalk-point correlation functions of their random solutions are derived. Parametric, deterministic PDEs for the laws of the random solutions are derived. Representations of the random solutions' laws on infinite-dimensional parameter spaces in terms of ‘generalized polynomial chaos’ (GPC) series are established. Recent results on the regularity of solutions of these parametric PDEs are presented. Convergence rates of bestN-term approximations, for adaptive stochastic Galerkin and collocation discretizations of the parametric, deterministic PDEs, are established. Sparse tensor products of hierarchical (multi-level) discretizations in physical space (and time), and GPC expansions in parameter space, are shown to converge at rates which are independent of the dimension of the parameter space. A convergence analysis of multi-level Monte Carlo (MLMC) discretizations of PDEs with random coefficients is presented. Sufficient conditions on the random inputs for superiority of sparse tensor discretizations over MLMC discretizations are established for linear elliptic, parabolic and hyperbolic PDEs with random coefficients.

The partition bundle of type A N−1 (2, 0) theory

Six-dimensional (2, 0) theory can be defined on a large class of six-manifolds endowed with some additional topological and geometric data (i.e. an orientation, a spin structure, a conformal structure, and an R-symmetry bundle with connection). We discuss the nature of the object that generalizes the partition function of a more conventional quantum theory. This object takes its values in a certain complex vector space, which fits together into the total space of a complex vector bundle (the `partition bundle') as the data on the six-manifold is varied in its infinite-dimensional parameter space. In this context, an important role is played by the middle-dimensional intermediate Jacobian of the six-manifold endowed with some additional data (i.e. a symplectic structure, a quadratic form, and a complex structure). We define a certain hermitian vector bundle over this finite-dimensional parameter space. The partition bundle is then given by the pullback of the latter bundle by the map from the parameter space related to the six-manifold to the parameter space related to the intermediate Jacobian.