Bayesian Experimental Design Research Articles

A Maximin Φp-Efficient Design for Multivariate Generalized Linear Models

Experimental designs for a generalized linear model (GLM) often depend on the specification of the model, including the link function, the predictors, and unknown parameters, such as the regression coefficients. To deal with uncertainties of these model specifications, it is important to construct optimal designs with high efficiency under such uncertainties. Existing methods such as Bayesian experimental designs often use prior distributions of model specifications to incorporate model uncertainties into the design criterion. Alternatively, one can obtain the design by optimizing the worst-case design efficiency with respect to uncertainties of model specifications. In this work, we propose a new Maximin $\Phi_p$-Efficient (or Mm-$\Phi_p$ for short) design which aims at maximizing the minimum $\Phi_p$-efficiency under model uncertainties. Based on the theoretical properties of the proposed criterion, we develop an efficient algorithm with sound convergence properties to construct the Mm-$\Phi_p$ design. The performance of the proposed Mm-$\Phi_p$ design is assessed through several numerical examples.

Bayesian D-Optimal Design of Experiments with Quantitative and Qualitative Responses

Systems with both quantitative and qualitative responses are widely encountered in many applications. Design of experiment methods are needed when experiments are conducted to study such systems. Classic experimental design methods are unsuitable here because they often focus on one type of response. In this paper, we develop a Bayesian D-optimal design method for experiments with one continuous and one binary response. Both noninformative and conjugate informative prior distributions on the unknown parameters are considered. The proposed design criterion has meaningful interpretations regarding the D-optimality for the models for both types of responses. An efficient point-exchange search algorithm is developed to construct the local D-optimal designs for given parameter values. Global D-optimal designs are obtained by accumulating the frequencies of the design points in local D-optimal designs, where the parameters are sampled from the prior distributions. The performances of the proposed methods are evaluated through two examples.

Bayesian Experimental Design for Efficient Sensor Placement in Two-Dimensional Electromagnetic Imaging

Bayesian Experimental Design for Efficient Sensor Placement in Two-Dimensional Electromagnetic Imaging

Discovering and forecasting extreme events via active learning in neural operators.

Extreme events in society and nature, such as pandemic spikes, rogue waves or structural failures, can have catastrophic consequences. Characterizing extremes is difficult, as they occur rarely, arise from seemingly benign conditions, and belong to complex and often unknown infinite-dimensional systems. Such challenges render attempts at characterizing them moot. We address each of these difficulties by combining output-weighted training schemes in Bayesian experimental design (BED) with an ensemble of deep neural operators. This model-agnostic framework pairs a BED scheme that actively selects data for quantifying extreme events with an ensemble of deep neural operators that approximate infinite-dimensional nonlinear operators. We show that not only does this framework outperform Gaussian processes, but that (1) shallow ensembles of just two members perform best; (2) extremes are uncovered regardless of the state of the initial data (that is, with or without extremes); (3) our method eliminates 'double-descent' phenomena; (4) the use of batches of suboptimal acquisition samples compared to step-by-step global optima does not hinder BED performance; and (5) Monte Carlo acquisition outperforms standard optimizers in high dimensions. Together, these conclusions form a scalable artificial intelligence (AI)-assisted experimental infrastructure that can efficiently infer and pinpoint critical situations across many domains, from physical to societal systems.

Improving extracellular vesicles production through a Bayesian optimization-based experimental design

With the growing demand and diversity of biological drugs, developing optimal processes for their accelerated production with minimal resource utilization is a pressing challenge. Typically, such optimization involves multiple target properties, such as production yield, biological activity, and product purity. Therefore, strategic experimental design techniques that can characterize the parameter space while simultaneously arriving at the optimal process satisfying multiple target properties are required. To achieve this, we propose the use of a multi-objective batch Bayesian optimization (MOBBO) algorithm and illustrate its successful application for the production of extracellular vesicles (EVs) from a 3D culture of mesenchymal stem cells (MSCs) considering three objectives, namely to maximize the vesicle-to-protein ratio, maximize the enzymatic activity of the MSC-EV protein CD73, and minimize the amount of calregulin impurities. We show that the optimal combination of the process parameters to address the intended objectives could be achieved with only 32 experiments. For the four parameters considered (i.e., microcarrier concentration, seeding density, centrifugation time, and impeller speed), this number of experiments is comparable to or lower than the classical design of experiments (DoE) and the traditional one-factor-at-a-time (OFAT) approach. We illustrate how the algorithm adaptively samples in the process parameter space, selectively excluding unfavorable regions, thus minimizing the number of experiments required to reach optimal conditions. Finally, we compare the obtained solutions to the literature data and present possible applications of the collected data for other modeling activities such as Quality by Design, process monitoring, control, and scale-up.

Bayesian Decision-Theoretic Design of Experiments Under an Alternative Model

Traditionally Bayesian decision-theoretic design of experiments proceeds by choosing a design to minimise expectation of a given loss function over the space of all designs. The loss function encapsulates the aim of the experiment, and the expectation is taken with respect to the joint distribution of all unknown quantities implied by the statistical model that will be fitted to observed responses. In this paper, an extended framework is proposed whereby the expectation of the loss is taken with respect to a joint distribution implied by an alternative statistical model. Motivation for this includes promoting robustness, ensuring computational feasibility and for allowing realistic prior specification when deriving a design. To aid in exploring the new framework, an asymptotic approximation to the expected loss under an alternative model is derived, and the properties of different loss functions are established. The framework is then demonstrated on a linear regression versus full-treatment model scenario, on estimating parameters of a non-linear model under model discrepancy and a cubic spline model under an unknown number of basis functions.

Efficient computation of temporal exceeding probability of ship responses in a random wave field

In this work, we develop a computational framework to efficiently quantify the temporal exceeding probability of ship responses in a random wave field generated by a narrow-band spectrum, i.e., the fraction of time that ship responses exceed a given threshold. In particular, we consider large thresholds so that the response exceedance needs to be treated as rare events. Our computational framework builds on the parameterization of wave field into wave groups and efficient sampling in the group parameter space through Bayesian experimental design (BED). Previous works following this framework exclusively studied extreme statistics of group-maximum response, i.e., the maximum response within a wave group, which is however not straightforward to interpret in practice (e.g., its value depends on the subjective definition of the wave group). In order to adapt the framework to the more robust measure by the temporal exceeding probability, novel developments need to be introduced to the two components of BED (on surrogate model and acquisition function), which are detailed in the paper. We validate our framework for a case of ship roll motion (without capsizing) calculated by a nonlinear roll equation, in terms of the agreement of our results to the true solution and the independence of our results to the criterion of defining groups. Finally, we demonstrate the coupling of the framework with CFD models to handle more realistic and general ship response problems.

Simplified Algorithms for Adaptive Experiment Design in Parameter Estimation.

Measurements to estimate parameters of a model are commonplace in the physical sciences, where the traditional approach to automation is to use a sequence of preselected settings followed by least-squares fitting of a model function to the data. This measure-then-fit approach is simple and effective and entirely appropriate for many applications but when measurement resources are limited, efficiency becomes more important. To increase efficiency, Bayesian experiment design allows measurement settings to be chosen adaptively based on accumulated data and utility, the predicted improvement in results as a function of settings. However, the calculation of utility has been judged too impractical for most applications. In this paper, we introduce computational methods and simplified algorithms that accelerate utility calculations by over an order of magnitude, with only slight degradation in measurement efficiency. The methods eliminate utility calculation as a barrier to practical application of efficient adaptive measurement.

Comparing Well and Geophysical Data for Temperature Monitoring Within a Bayesian Experimental Design Framework

AbstractTemperature logs are an important tool in the geothermal industry. Temperature measurements from boreholes are used for exploration, system design, and monitoring. The number of observations, however, is not always sufficient to fully determine the temperature field or explore the entire parameter space of interest. Drilling in the best locations is still difficult and expensive. It is therefore critical to optimize the number and location of boreholes. Due to its higher spatial resolution and lower cost, four‐dimensional (4D) temperature field monitoring via time‐lapse Electrical Resistivity Tomography has been investigated as a potential alternative. We use Bayesian Evidential Learning (BEL), a Monte Carlo‐based training approach, to optimize the design of a 4D temperature field monitoring experiment. We demonstrate how BEL can take into account various data source combinations (temperature logs combined with geophysical data) in the Bayesian optimal experimental design (BOED). To determine the optimal data source combination, we use the Root Mean Squared Error of the predicted target in the low dimensional latent space where BEL is solving the prediction problem. The parameter estimates are accurate enough to use in BOED. Furthermore, the method is not limited to monitoring temperature fields and can be applied to other similar experimental design problems. The method is computationally efficient and requires little training data. For the considered optimal design problem, a training set of only 200 samples and a test set of 50 samples is sufficient.

Modeling intensification decisions in the Kilombero Valley floodplain: A Bayesian belief network approach

AbstractThe Kilombero Valley floodplain in Tanzania is a major agricultural area. Government initiatives and projects supported by international funding have long sought to boost productivity. Due to increasing population pressure, smallholder farmers are forced to increase their output. Nevertheless, the level of intensification is still lower than what is considered necessary to increase production and support smallholder livelihoods significantly. This article aims to better understand farmers’ intensification choices and their interdependent determinants. We propose a novel modeling approach for identifying determinants of intensification and their interrelationships by combining a Bayesian belief network (BBN), experimental design, and multivariate regression trees. Our approach complements existing lower‐dimensional statistical models by considering uncertainty and providing an easily updatable model structure. The BBN is constructed and calibrated using data from a survey of 304 farm households. Our findings show how the data‐driven BBN approach can be used to identify variables that influence farmers’ decision to choose one technique over another. Furthermore, the most important drivers vary widely, depending on the intensification options being considered.

Real-Time Adaptive Sensing of Nuclear Spins by a Single-Spin Quantum Sensor

Quantum sensing is considered to be one of the most promising subfields of quantum information to deliver practical quantum advantages in real-world applications. However, its impressive capabilities, including high sensitivity, are often hindered by the limited quantum resources available. Here, we incorporate the expected information gain (EIG) and techniques such as accelerated computation into Bayesian experimental design (BED) in order to use quantum resources more efficiently. A simulated nitrogen-vacancy center in diamond is used to demonstrate real-time operation of the BED. Instead of heuristics, the EIG is used to choose optimal control parameters in real-time. Moreover, combining the BED with accelerated computation and asynchronous operations, we find that up to a tenfold speed-up in absolute time cost can be achieved in sensing multiple surrounding C13 nuclear spins. Our work explores the possibilities of applying the EIG to BED-based quantum-sensing tasks and provides techniques useful to integrate BED into more generalized quantum sensing systems.

Small-noise approximation for Bayesian optimal experimental design with nuisance uncertainty

Calculating the expected information gain in optimal Bayesian experimental design typically relies on nested Monte Carlo sampling. When the model also contains nuisance parameters, which are parameters that contribute to the overall uncertainty of the system but are of no interest in the Bayesian design framework, this introduces a second inner loop. We propose and derive a small-noise approximation for this additional inner loop. The computational cost of our method can be further reduced by applying a Laplace approximation to the remaining inner loop. Thus, we present two methods, the small-noise double-loop Monte Carlo and small-noise Monte Carlo Laplace methods. Moreover, we demonstrate that the total complexity of these two approaches remains comparable to the case without nuisance uncertainty. To assess the efficiency of these methods, we present three examples, and the last example includes the partial differential equation for the electrical impedance tomography experiment for composite laminate materials.

Bayesian probabilistic propagation of hybrid uncertainties: Estimation of response expectation function, its variable importance and bounds

Uncertainties existing in physical and engineering systems can be characterized by different kinds of mathematical models according to their respective features. However, efficient propagation of hybrid uncertainties via an expensive-to-evaluate computer simulator is still a computationally challenging task. In this contribution, estimation of response expectation function (REF), its variable importance and bounds under hybrid uncertainties in the form of precise probability models, parameterized probability-box models and interval models is investigated through a Bayesian approach. Specifically, a new method, termed “Parallel Bayesian Quadrature Optimization” (PBQO), is developed. The method starts by treating the REF estimation as a Bayesian probabilistic integration (BPI) problem with a Gaussian process (GP) prior, which in turn implies a GP posterior for the REF. Then, one acquisition function originally developed in BPI and other two in Bayesian global optimization are introduced for Bayesian experimental designs. Besides, an innovative strategy is also proposed to realize multi-point selection at each iteration. Overall, a novel advantage of PBQO is that it is capable of yielding the REF, its variable importance and bounds simultaneously via a pure single-loop procedure allowing for parallel computing. Three numerical examples are studied to demonstrate the performance of the proposed method over some existing methods.

Reinforcement Learning-Based Sequential Batch-Sampling for Bayesian Optimal Experimental Design

AbstractEngineering problems that are modeled using sophisticated mathematical methods or are characterized by expensive-to-conduct tests or experiments are encumbered with limited budget or finite computational resources. Moreover, practical scenarios in the industry, impose restrictions, based on logistics and preference, on the manner in which the experiments can be conducted. For example, material supply may enable only a handful of experiments in a single-shot or in the case of computational models one may face significant wait-time based on shared computational resources. In such scenarios, one usually resorts to performing experiments in a manner that allows for maximizing one’s state-of-knowledge while satisfying the above-mentioned practical constraints. Sequential design of experiments (SDOE) is a popular suite of methods that have yielded promising results in recent years across different engineering and practical problems. A common strategy that leverages Bayesian formalism is the Bayesian SDOE, which usually works best in the one-step-ahead or myopic scenario of selecting a single experiment at each step of a sequence of experiments. In this work, we aim to extend the SDOE strategy, to query the experiment or computer code at a batch of inputs. To this end, we leverage deep reinforcement learning (RL)-based policy gradient methods, to propose batches of queries that are selected taking into account the entire budget in hand. The algorithm retains the sequential nature, inherent in the SDOE while incorporating elements of reward based on task from the domain of deep RL. A unique capability of the proposed methodology is its ability to be applied to multiple tasks, for example, optimization of a function, once its trained. We demonstrate the performance of the proposed algorithm on a synthetic problem and a challenging high-dimensional engineering problem.

Tractable global solutions to chance-constrained Bayesian optimal experiment design for arbitrary prior and noise distributions

Optimal experiment design (OED) aims to optimize the information content of experimental observations by designing the experimental conditions. In Bayesian OED for parameter estimation, the design selection is based on an expected utility metric that accounts for the joint probability distribution of the uncertain parameters and the observations. This work presents solution methods for two approximate formulations of the Bayesian OED problem based on Kullback–Leibler divergence for the particular case of Gaussian prior and observation noise distributions and the general case of arbitrary prior distributions and arbitrary observation noise distributions when the observation noise corresponds to arbitrary functions of the states and random variables with an arbitrary multivariate distribution. The proposed methods also allow satisfying path constraints with a specified probability. The solution approach relies on the reformulation of the approximate Bayesian OED problem as an optimal control problem (OCP), for which a parsimonious input parameterization is adopted to reduce the number of decision variables. An efficient global solution method for OCPs via sum-of-squares polynomials and parallel computing is then applied, which is based on approximating the cost of the OCP by a polynomial function of the decision variables and solving the resulting polynomial optimization problem to global optimality in a tractable way via semidefinite programming. It is established that the difference between the cost obtained by solving the polynomial optimization problem and the globally optimal cost of the OCP is bounded and depends on the polynomial approximation error.

Edge-Promoting Adaptive Bayesian Experimental Design for X-ray Imaging

This work considers sequential edge-promoting Bayesian experimental design for (discretized) linear inverse problems, exemplified by X-ray tomography. The process of computing a total variation type reconstruction of the absorption inside the imaged body via lagged diffusivity iteration is interpreted in the Bayesian framework. Assuming a Gaussian additive noise model, this leads to an approximate Gaussian posterior with a covariance structure that contains information on the location of edges in the posterior mean. The next projection geometry is then chosen through A-optimal Bayesian design, which corresponds to minimizing the trace of the updated posterior covariance matrix that accounts for the new projection. Two and three-dimensional numerical examples based on simulated data demonstrate the functionality of the introduced approach.

Optimized arrays for electrical resistivity tomography survey using Bayesian experimental design

Electrical resistivity tomography (ERT) is broadly used to characterize and monitor subsurface processes because of its simplicity in data acquisition and modeling. The resolution of ERT is controlled by the position of the electrodes and the measurement protocol. As a result, there has been increasing interest in optimizing ERT surveys for higher resolution. The most widely used “Compare-R” method optimizes the ERT survey by calculating the updates of possible measurement configurations to the resolution matrix. The computational burden of the Compare-R method becomes overwhelming due to the computational cost of the estimated resolution matrix with an increasing number of electrodes. We have developed a new ERT survey design methodology for a target of interest based on Bayesian experimental design. The computational challenge is alleviated through two main strategies. First, in our method, the resolution matrix is not necessary and the optimized measurements are determined using higher relative entropy, which represents the maximization of the expected information gain. Second, the simulated measurements of four-electrode configurations to calculate relative entropy are reconstructed from two-electrode configurations using the principle of superposition. Static and time-lapse ERT synthetic surveys are used to test the performance of our method. The robustness of the optimized measurement configuration is evaluated by adding different levels of noise. Results find that the Bayesian experimental design can acquire optimized measurement configurations with similar resolution compared to the Compare-R method. However, the Bayesian optimized measurement configurations are more robust to noise and the computational cost is reduced by up to 38% and 89% for 2D and 3D cases, respectively. This method has great potential in optimized ERT survey design, especially for high-resolution three dimensions or crosswell ERT surveys.

Sequential Bayesian experimental design for estimation of extreme-event probability in stochastic input-to-response systems

We consider an input-to-response (ItR) system characterized by (1) parameterized input with a known probability distribution and (2) stochastic ItR function with heteroscedastic randomness. Our purpose is to efficiently quantify the extreme response probability when the ItR function is expensive to evaluate. The problem setup arises often in physics and engineering problems, with randomness in ItR coming from either intrinsic uncertainties (say, as a solution to a stochastic equation) or additional (critical) uncertainties that are not incorporated in a low-dimensional input parameter space (as a result of dimension reduction applied to the original high-dimensional input space). To reduce the required sampling numbers, we develop a sequential Bayesian experimental design method leveraging the variational heteroscedastic Gaussian process regression (VHGPR) to account for the stochastic ItR, along with a new criterion to select the next-best samples sequentially. The validity of our new method is first tested in two synthetic problems with the stochastic ItR functions defined artificially. Finally, we demonstrate the application of our method to an engineering problem of estimating the extreme ship motion probability in irregular waves, where the uncertainty in ItR naturally originates from standard wave group parameterization, which reduces the original high-dimensional wave field into a two-dimensional parameter space.

Stochastic Learning Approach for Binary Optimization: Application to Bayesian Optimal Design of Experiments

We present a novel stochastic approach to binary optimization for optimal experimental design (OED) for Bayesian inverse problems governed by mathematical models such as partial differential equations. The OED utility function, namely, the regularized optimality criterion, is cast into a stochastic objective function in the form of an expectation over a multivariate Bernoulli distribution. The probabilistic objective is then solved by using a stochastic optimization routine to find an optimal observational policy. The proposed approach is analyzed from an optimization perspective and also from a machine learning perspective with correspondence to policy gradient reinforcement learning. The approach is demonstrated numerically by using an idealized two-dimensional Bayesian linear inverse problem, and validated by extensive numerical experiments carried out for sensor placement in a parameter identification setup.

Two-stage Bayesian experimental design optimization for measuring soil–water characteristic curve

Two-stage Bayesian experimental design optimization for measuring soil–water characteristic curve