Optimal Bayesian Experimental Design Research Articles

Small-noise approximation for Bayesian optimal experimental design with nuisance uncertainty

Calculating the expected information gain in optimal Bayesian experimental design typically relies on nested Monte Carlo sampling. When the model also contains nuisance parameters, which are parameters that contribute to the overall uncertainty of the system but are of no interest in the Bayesian design framework, this introduces a second inner loop. We propose and derive a small-noise approximation for this additional inner loop. The computational cost of our method can be further reduced by applying a Laplace approximation to the remaining inner loop. Thus, we present two methods, the small-noise double-loop Monte Carlo and small-noise Monte Carlo Laplace methods. Moreover, we demonstrate that the total complexity of these two approaches remains comparable to the case without nuisance uncertainty. To assess the efficiency of these methods, we present three examples, and the last example includes the partial differential equation for the electrical impedance tomography experiment for composite laminate materials.

Optimal Bayesian experimental design for electrical impedance tomography in medical imaging

Optimal design of electronic devices such as sensors is essential since it results in more accurate output at the shortest possible time. In this work, we develop optimal Bayesian inversion for electrical impedance tomography (EIT) technology in order to improve the quality of medical images generated by EIT and to put this promising imaging technology into practice. We optimize Bayesian experimental design by maximizing the expected information gain in the Bayesian inversion process in order to design optimal experiments and obtain the most informative data about the unknown parameters. We present optimal experimental designs including optimal frequency and optimal electrode configuration, all of which result in the most accurate estimation of the unknown quantities to date and high-resolution EIT medical images, which are crucial for diagnostic purposes. Numerical results show the efficiency of the proposed optimal Bayesian inversion method for the EIT inverse problem.

Optimal Bayesian experimental design for subsurface flow problems

Optimal Bayesian design techniques provide an estimate for the best parameters of an experiment in order to maximize the value of measurements prior to the actual collection of data. In other words, these techniques explore the space of possible observations and determine an experimental setup that produces maximum information about the system parameters on average. Generally, optimal Bayesian design formulations result in multiple high-dimensional integrals that are difficult to evaluate without incurring significant computational costs as each integration point corresponds to solving a coupled system of partial differential equations.In the present work, we propose a novel approach for development of polynomial chaos expansion (PCE) surrogate model for the design utility function. In particular, we demonstrate how the orthogonality of PCE basis polynomials can be utilized in order to replace the expensive integration over the space of possible observations by direct construction of PCE approximation for the expected information gain. This novel technique enables the derivation of a reasonable quality response surface for the targeted objective function with a computational budget comparable to several single-point evaluations. Therefore, the proposed technique reduces dramatically the overall cost of optimal Bayesian experimental design. We evaluate this alternative formulation utilizing PCE on few numerical test cases with various levels of complexity to illustrate the computational advantages of the proposed approach.

An induced natural selection heuristic for finding optimal Bayesian experimental designs

Bayesian optimal experimental design has immense potential to inform the collection of data so as to subsequently enhance our understanding of a variety of processes. However, a major impediment is the difficulty in evaluating optimal designs for problems with large, or high-dimensional, design spaces. An efficient search heuristic suitable for general optimisation problems, with a particular focus on optimal Bayesian experimental design problems, is proposed. The heuristic evaluates the objective (utility) function at an initial, randomly generated set of input values. At each generation of the algorithm, input values are “accepted” if their corresponding objective (utility) function satisfies some acceptance criteria, and new inputs are sampled about these accepted points. The new algorithm is demonstrated by evaluating the optimal Bayesian experimental designs for the previously considered death, pharmacokinetic and logistic regression models. Comparisons to the current “gold-standard” method are given to demonstrate the proposed algorithm as a computationally-efficient alternative for moderately-large design problems (i.e., up to approximately 40-dimensions).

Fast Bayesian experimental design: Laplace-based importance sampling for the expected information gain

In calculating expected information gain in optimal Bayesian experimental design, the computation of the inner loop in the classical double-loop Monte Carlo requires a large number of samples and suffers from underflow if the number of samples is small. These drawbacks can be avoided by using an importance sampling approach. We present a computationally efficient method for optimal Bayesian experimental design that introduces importance sampling based on the Laplace method to the inner loop. We derive the optimal values for the method parameters in which the average computational cost is minimized for a specified error tolerance. We use three numerical examples to demonstrate the computational efficiency of our method compared with the classical double-loop Monte Carlo, and a single-loop Monte Carlo method that uses the Laplace approximation of the return value of the inner loop. The first demonstration example is a scalar problem that is linear in the uncertain parameter. The second example is a nonlinear scalar problem. The third example deals with the optimal sensor placement for an electrical impedance tomography experiment to recover the fiber orientation in laminate composites.

Optimal Bayesian Experimental Design for Models with Intractable Likelihoods Using Indirect Inference Applied to Biological Process Models

This paper addresses the problem of determining optimal designs for biological process models with intractable likelihoods, with the goal of parameter inference. The Bayesian approach is to choose a design that maximises the mean of a utility, and the utility is a function of the posterior distribution. Therefore, its estimation requires likelihood evaluations. However, many problems in experimental design involve models with intractable likelihoods, that is, likelihoods that are neither analytic nor can be computed in a reasonable amount of time. We propose a novel solution using indirect inference (II), a well established method in the literature, and the Markov chain Monte Carlo (MCMC) algorithm of Müller et al. (2004). Indirect inference employs an auxiliary model with a tractable likelihood in conjunction with the generative model, the assumed true model of interest, which has an intractable likelihood. Our approach is to estimate a map between the parameters of the generative and auxiliary models, using simulations from the generative model. An II posterior distribution is formed to expedite utility estimation. We also present a modification to the utility that allows the Müller algorithm to sample from a substantially sharpened utility surface, with little computational effort. Unlike competing methods, the II approach can handle complex design problems for models with intractable likelihoods on a continuous design space, with possible extension to many observations. The methodology is demonstrated using two stochastic models; a simple tractable death process used to validate the approach, and a motivating stochastic model for the population evolution of macroparasites.

Optimal Bayesian Experimental Design for Priors of Compact Support with Application to Shock‐Tube Experiments for Combustion Kinetics

SummaryThe analysis of reactive systems in combustion science and technology relies on detailed models comprising many chemical reactions that describe the conversion of fuel and oxidizer into products and the formation of pollutants. Shock‐tube experiments are a convenient setting for measuring the rate parameters of individual reactions. The temperature, pressure, and concentration of reactants are chosen to maximize the sensitivity of the measured quantities to the rate parameter of the target reaction. In this study, we optimize the experimental setup computationally by optimal experimental design in a Bayesian framework. We approximate the posterior probability density functions (pdf) using truncated Gaussian distributions in order to account for the bounded domain of the uniform prior pdf of the parameters. The underlying Gaussian distribution is obtained in the spirit of the Laplace method, more precisely, the mode is chosen as the maximum a posteriori (MAP) estimate, and the covariance is chosen as the negative inverse of the Hessian of the misfit function at the MAP estimate. The model related entities are obtained from a polynomial surrogate. The optimality, quantified by the information gain measures, can be estimated efficiently by a rejection sampling algorithm against the underlying Gaussian probability distribution, rather than against the true posterior. This approach offers a significant error reduction when the magnitude of the invariants of the posterior covariance are comparable with the size of the bounded domain of the prior. We demonstrate the accuracy and superior computational efficiency of our method for shock‐tube experiments aiming to measure the model parameters of a key reaction, which is part of the complex kinetic network describing the hydrocarbon oxidation. In the experiments, the initial temperature and fuel concentration are optimized with respect to the expected information gain in the estimation of the parameters of the target reaction rate. We show that the expected information gain surface can change its “shape” dramatically according to the level of noise introduced into the synthetic data. The information that can be extracted from the data saturates as a logarithmic function of the number of experiments, and few experiments are needed when they are conducted at the optimal experimental design conditions. Furthermore, inversion of the legacy data indicates the validity and robustness of our designs. Copyright © 2016 John Wiley & Sons, Ltd.

A stochastic programming approach for the Bayesian experimental design of nonlinear systems

Several approaches for the Bayesian design of experiments have been proposed in the literature (e.g., D-optimal, E-optimal, A-optimal designs). Most of these approaches assume that the available prior knowledge is represented by a normal probability distribution. In addition, most nonlinear design approaches involve assuming normality of the posterior distribution and approximate its variance using the expected Fisher information matrix. In order to be able to relax these assumptions, we address and generalize the problem by using a stochastic programming formulation. Specifically, the optimal Bayesian experimental design is mathematically posed as a three-stage stochastic program, which is then discretized using a scenario based approach. Given the prior probability distribution, a Smolyak rule (sparse-grids) is used for the selection of scenarios. Two retrospective case studies related to population pharmacokinetics are presented. The benefits and limitations of the proposed approach are demonstrated by comparing the numerical results to those obtained by implementing a more exhaustive experimentation and the D-optimal design.

Simulation-based optimal Bayesian experimental design for nonlinear systems

The optimal selection of experimental conditions is essential to maximizing the value of data for inference and prediction, particularly in situations where experiments are time-consuming and expensive to conduct. We propose a general mathematical framework and an algorithmic approach for optimal experimental design with nonlinear simulation-based models; in particular, we focus on finding sets of experiments that provide the most information about targeted sets of parameters.Our framework employs a Bayesian statistical setting, which provides a foundation for inference from noisy, indirect, and incomplete data, and a natural mechanism for incorporating heterogeneous sources of information. An objective function is constructed from information theoretic measures, reflecting expected information gain from proposed combinations of experiments. Polynomial chaos approximations and a two-stage Monte Carlo sampling method are used to evaluate the expected information gain. Stochastic approximation algorithms are then used to make optimization feasible in computationally intensive and high-dimensional settings. These algorithms are demonstrated on model problems and on nonlinear parameter inference problems arising in detailed combustion kinetics.

Maximum Entropy Sampling and Optimal Bayesian Experimental Design

Summary When Shannon entropy is used as a criterion in the optimal design of experiments, advantage can be taken of the classical identity representing the joint entropy of parameters and observations as the sum of the marginal entropy of the observations and the preposterior conditional entropy of the parameters. Following previous work in which this idea was used in spatial sampling, the method is applied to standard parameterized Bayesian optimal experimental design. Under suitable conditions, which include non-linear as well as linear regression models, it is shown in a few steps that maximizing the marginal entropy of the sample is equivalent to minimizing the preposterior entropy, the usual Bayesian criterion, thus avoiding the use of conditional distributions. It is shown using this marginal formulation that under normality assumptions every standard model which has a two-point prior distribution on the parameters gives an optimal design supported on a single point. Other results include a new asymptotic formula which applies as the error variance is large and bounds on support size.

Electrical Impedance Tomography

t. This paper surveys some of the work our group has done in electrical impedance tomography.

081026 (M10) Maximum entropy sampling and optimal Bayesian experimental design: Sebastiani P., Wynn H.P., Department of Actuarial Science and Statistics, City University, London, University of Warwick, Warwick, City University, Statistical Research Paper No. 3, 1996

081026 (M10) Maximum entropy sampling and optimal Bayesian experimental design: Sebastiani P., Wynn H.P., Department of Actuarial Science and Statistics, City University, London, University of Warwick, Warwick, City University, Statistical Research Paper No. 3, 1996

Bayesian optimal experimental design for treatment—control comparisons in the presence of two-way heterogeneity

Optimal Bayesian experimental design is derived for the one-way analysis of variance in the presence of two-way heterogeneity. The design is optimal for the estimation of the treatment-control contrasts, minimizing their posterior expected squared error loss. The prior knowledge of the treatment and blocking effects is modelled by Normal distribution, the treatments being exchangeable. A method for the derivation of the optimal design for any covariance structure of the blocking effects is given. For a special interesting covariance matrix of the blocking effects, the optimal design is derived.

Correction: Optimal Bayesian Experimental Design for Linear Models

Correction: Optimal Bayesian Experimental Design for Linear Models

Optimal Bayesian Experimental Design for Linear Models

Optimal Bayesian experimental designs for estimation and prediction in linear models are discussed. The designs are optimal for estimating a linear combination of the regression parameters $\mathbf{c}^T\theta$ or prediction at a point where the expected response is $\mathbf{c}^T\mathbf{\theta}$ under squared error loss. A distribution on $\mathbf{c}$ is introduced to represent the interest in particular linear combinations of the parameters. In the usual notation for linear models minimizing the preposterior expected loss leads to minimizing the quantity $\mathrm{tr}\psi(R + XX^T)^{-1}$. The matrix $\psi$ is defined to be $E(\mathbf{cc}^T)$ and the matrix $R$ is the prior precision matrix of $\theta$. A geometric interpretation of the optimal designs is given which leads to a parallel of Elfving's theorem for $\mathbf{c}$-optimality. A bound is given for the minimum number of points at which it is necessary to take observations. Some examples of optimal Bayesian designs are given and optimal designs for prediction in polynomial regression are derived. The optimality of rounding non-integer designs to integer designs is discussed.