Experimental Design Problem Research Articles

Optimal design of mixture experiments for general blending models

Mixture models of the Scheffé polynomial class are standard in several scientific fields. For these models there is a vast literature on the optimal design of experiments to provide good estimates of the parameters with the use of minimal resources. Contrarily, the optimal design of experiments for general blending models, extending the class of Becker, have not been systematically addressed. Nevertheless, there are practical examples where the models relating the response variables, the parameters and the factors including nonlinear blending effects fall into a general form. We propose a general formulation to find continuous and exact D– and A–optimal designs for general blending models. First, we consider designs to estimate the regression coefficients, and then extend the formulations to find locally optimal continuous designs for estimating both the coefficients and the power exponents. The treatment relies on converting the Optimal Experimental Design (OED) problem into an optimization problem of the Nonlinear Programming (or Mixed Integer Nonlinear Programming) class. We apply this approach to quadratic and special cubic general blending models of the H2 class of polynomials introduced by Becker, and to three examples of practical interest in combustion science and in the characterization of fuel properties.

A branch-and-bound algorithm for the exact optimal experimental design problem

We discuss a generalisation of the approximate optimal experimental design problem, in which the weight of each regression point needs to stay in a closed interval. We work with Kiefer’s optimality criteria which include the well-known D- and A-optimality as special cases. We propose a first-order algorithm for the generalised problem that redistributes the weights of two regression points in each iteration. We develop a branch-and-bound algorithm for exact optimal experimental design problems under Kiefer’s criteria where the subproblems in the search tree are equivalent to the generalized approximate design problem, and therefore, can be solved efficiently by the first-order method. We observe that our branch-and-bound algorithm is favourable to a popular exchange heuristic for certain problem instances.

Designing experiments informed by observational studies

AbstractThe increasing availability of passively observed data has yielded a growing interest in “data fusion” methods, which involve merging data from observational and experimental sources to draw causal conclusions. Such methods often require a precarious tradeoff between the unknown bias in the observational dataset and the often-large variance in the experimental dataset. We propose an alternative approach, which avoids this tradeoff: rather than using observational data for inference, we use it to design a more efficient experiment. We consider the case of a stratified experiment with a binary outcome and suppose pilot estimates for the stratum potential outcome variances can be obtained from the observational study. We extend existing results to generate confidence sets for these variances, while accounting for the possibility of unmeasured confounding. Then, we pose the experimental design problem as a regret minimization problem subject to the constraints imposed by our confidence sets. We show that this problem can be converted into a concave maximization and solved using conventional methods. Finally, we demonstrate the practical utility of our methods using data from the Women’s Health Initiative.

Optimal experimental design for linear time invariant state–space models

The linear time invariant state–space model representation is common to systems from several areas ranging from engineering to biochemistry. We address the problem of systematic optimal experimental design for this class of model. We consider two distinct scenarios: (i) steady-state model representations and (ii) dynamic models described by discrete-time representations. We use our approach to construct locally D-optimal designs by incorporating the calculation of the determinant of the Fisher Information Matrix and the parametric sensitivity computation in a Nonlinear Programming formulation. A global optimization solver handles the resulting numerical problem. The Fisher Information Matrix at convergence is used to determine model identifiability. We apply the methodology proposed to find approximate and exact optimal experimental designs for static and dynamic experiments for models representing a biochemical reaction network where the experimental purpose is to estimate kinetic constants.

Experimental design and data processing in a distance navigation problem for position of the object location

A problem of optimizing the configuration of a navigation measuring system is considered in terms of the experimental design using a distance navigation problem for position of the object location. It is shown that the stated problem is equivalent to the problem of A-optimal experimental design for a regression function (nonlinear in parameters) and can be reduced to a trigonometric model. The response function, Fisher’s information and the sensitivity factor of the navigation system in case of two and three beacons and correlated measurements are presented in an explicit form. Using the equivalence theorem for A-criterion in the case of two-dimensional (plane) distance problem we confirm again the Barabanovs’s result that matrixes of A-optimal designs are the Kolmogorov – Maltsev matrixes. A similar result holds for the D-optimality criterion in the considered case. The effect of the measurement correlation in a distance navigation problem with two and three reference points is considered. The formulas for the sensitivity factors expressed in terms of bearings on the reference points and intersection angle of object are derived. In addition to a problem of optimizing the network configuration, the data processing problem in two-dimensional distance navigation problem with two reference points is also considered. The location of the object is determined in two ways, i.e., using the geometrical method and method of resultants. In the first method the solution of a distance navigation problem comes to the consideration of two independent quadratic equations for determination of the first and the second coordinates of the object. The equations are obtained in the explicit form. The second method also leads to two quadratic equations for determination of the object location. This is an option of the exclusion method which provides for an explicit form of conditions ensuring the solution of the considered problem for determination of the object location. Examples are considered that confirm the stated conclusions.

Sequential Bayesian optimal experimental design for structural reliability analysis

Structural reliability analysis is concerned with estimation of the probability of a critical event taking place, described by P(g(mathbf{X} ) le 0) for some n-dimensional random variable mathbf{X} and some real-valued function g. In many applications the function g is practically unknown, as function evaluation involves time consuming numerical simulation or some other form of experiment that is expensive to perform. The problem we address in this paper is how to optimally design experiments, in a Bayesian decision theoretic fashion, when the goal is to estimate the probability P(g(mathbf{X} ) le 0) using a minimal amount of resources. As opposed to existing methods that have been proposed for this purpose, we consider a general structural reliability model given in hierarchical form. We therefore introduce a general formulation of the experimental design problem, where we distinguish between the uncertainty related to the random variable mathbf{X} and any additional epistemic uncertainty that we want to reduce through experimentation. The effectiveness of a design strategy is evaluated through a measure of residual uncertainty, and efficient approximation of this quantity is crucial if we want to apply algorithms that search for an optimal strategy. The method we propose is based on importance sampling combined with the unscented transform for epistemic uncertainty propagation. We implement this for the myopic (one-step look ahead) alternative, and demonstrate the effectiveness through a series of numerical experiments.

Optimum degradation test plan under model mis-specification for Wiener and gamma processes

Degradation tests have been used for the purpose of assessing reliability information. In this paper, we consider a degradation test design problem under a statistical model mis-specification scenario. The Wiener and gamma processes are considered in this research. A gamma process is suitable for describing degradation paths that exhibit monotone behavior. However, if one fits a Wiener process model to the data, the resulting statistical inferences may be affected. Tsai, Tseng, and Balakrishnan studied the effect on the estimated mean time to failure. However, the experimental design problem was not discussed. A lack of the explicit functional form of the estimation variances makes it difficult to find degradation plans that can improve test efficiency. In this paper, functional forms of optimal degradation test plans are proposed under model mis-specification. Furthermore, a weighted ratio objective function considering the prior probability of the true model is used to find robust test plans for practical use. The results from a numerical example show that, by using an appropriate degradation test plan, the estimation variance can be reduced, and the test efficiency can be improved. A simulation study is conducted to investigate the performance of the parameter estimates when the sample size is small.

Robust optimal identification experiment design for multisine excitation

In least costly experiment design, the optimal spectrum of an identification experiment is determined in such a way that the cost of the experiment is minimized under some accuracy constraint on the identified parameter vector. Like all optimal experiment design problems, this optimization problem depends on the unknown true system, which is generally replaced by an initial estimate. One important consequence of this is that we can underestimate the actual cost of the experiment and that the accuracy of the identified model can be lower than desired. Here, based on an a-priori uncertainty set for the true system, we propose a convex optimization approach that allows to prevent these issues from happening. We do this when the to-be-determined spectrum is the one of a multisine signal.

Bayesian experimental design for CO2 sequestration with leakage risk assessment

PurposeThe author examine the sequestration of CO2 in abandoned geological formations where leakages are permitted up to only a certain threshold to meet the international CO2 emissions standards. Technically, the author address a Bayesian experimental design problem to optimally mitigate uncertainties and to perform risk assessment on a CO2 sequestration model, where the parameters to be inferred are random subsurface properties while the quantity of interest is desired to be kept within safety margins.Design/methodology/approachThe author start with a probabilistic formulation of learning the leak-age rate, and the author later relax it to a Bayesian experimental design of learning the formations geo-physical properties. The injection rate is the design parameter, and the learned properties are used to estimate the leakage rate by means of a nonlinear operator. The forward model governs a two-phase two-component flow in a porous medium with no solubility of CO2 in water. The Laplace approximation is combined with Monte Carlo sampling to estimate the expectation of the Kullback–Leibler divergence that stands for the objective function.FindingsDifferent scenarios, of confining CO2 while measuring the risk of harmful leakages, are analyzed numerically. The efficiency of the inversion of the CO2 leakage rate improves with the injection rate as great improvements, in terms of the accuracy of the estimation of the formation properties, are noticed. However, this study shows that those results do not imply in any way that the learned value of the CO2 leakage should exhibit the same behavior. Also this study enhances the implementation of CO2 sequestrations by extending the duration given by the reservoir capacity, controlling the injection while the emissions remain in agreement with the international standards.Originality/valueUncertainty quantification of the reservoir properties is addressed. Nonlinear goal-oriented inverse problem, for the estimation of the leakage rate, is known to be very challenging. This study presents a relaxation of the probabilistic design of learning the leakage rate to the Bayesian experimental design of learning the reservoir geophysical properties.

Dual optimal design and the Christoffel–Darboux polynomial

The purpose of this short note is to show that the Christoffel-Darboux polynomial, useful in approximation theory and data science, arises naturally when deriving the dual to the problem of semi-algebraic D-optimal experimental design in statistics. It uses only elementary notions of convex analysis. Geometric interpretations and algorithmic consequences are mentioned.

Optimal Experimental Design for Uncertain Systems Based on Coupled Differential Equations

We consider the optimal experimental design problem for an uncertain Kuramoto model, which consists of N interacting oscillators described by coupled ordinary differential equations. The objective is to design experiments that can effectively reduce the uncertainty present in the coupling strengths between the oscillators, thereby minimizing the cost of robust control of the uncertain Kuramoto model. We demonstrate the importance of quantifying the operational impact of the potential experiments in designing optimal experiments.

Approximation Algorithms forD-optimal Design

Experimental design is a classical statistics problem, and its aim is to estimate an unknown vector from linear measurements where a Gaussian noise is introduced in each measurement. For the combinatorial experimental design problem, the goal is to pick a subset of experiments so as to make the most accurate estimate of the unknown parameters. In this paper, we will study one of the most robust measures of error estimation—the D-optimality criterion, which corresponds to minimizing the volume of the confidence ellipsoid for the estimation error. The problem gives rise to two natural variants depending on whether repetitions of experiments are allowed or not. We first propose an approximation algorithm with a 1/e-approximation for the D-optimal design problem with and without repetitions, giving the first constant-factor approximation for the problem. We then analyze another sampling approximation algorithm and prove that it is asymptotically optimal. Finally, for D-optimal design with repetitions, we study a different algorithm proposed by the literature and show that it can improve this asymptotic approximation ratio. All the sampling algorithms studied in this paper are shown to admit polynomial-time deterministic implementations.

A Multiobjective LQR Synthesis Approach to Dual Control for Uncertain Plants

This letter proposes a dual control finite horizon LQR synthesis procedure for unknown systems characterized by mean and covariance estimates. The optimized policy comprises time-varying state-feedback and dithering components, and the control problem is framed as a multiobjective synthesis which seeks a balance between exploitation and exploration costs. It is shown that classic experiment design problems can be recast in this framework by replacing the exploitation cost with an information reward. Numerical examples demonstrate the different dual control trade-offs on plants with different properties.

A Theory of Experimenters: Robustness, Randomization, and Balance

This paper studies the problem of experiment design by an ambiguity-averse decision-maker who trades off subjective expected performance against robust performance guarantees. This framework accounts for real-world experimenters’ preference for randomization. It also clarifies the circumstances in which randomization is optimal: when the available sample size is large and robustness is an important concern. We apply our model to shed light on the practice of rerandomization, used to improve balance across treatment and control groups. We show that rerandomization creates a trade-off between subjective performance and robust performance guarantees. However, robust performance guarantees diminish very slowly with the number of rerandomizations. This suggests that moderate levels of rerandomization usefully expand the set of acceptable compromises between subjective performance and robustness. Targeting a fixed quantile of balance is safer than targeting an absolute balance objective. (JEL C90, D81)

Parallel Gaussian Process Surrogate Bayesian Inference with Noisy Likelihood Evaluations

We consider Bayesian inference when only a limited number of noisy log-likelihood evaluations can be obtained. This occurs for example when complex simulator-based statistical models are fitted to data, and synthetic likelihood (SL) method is used to form the noisy log-likelihood estimates using computationally costly forward simulations. We frame the inference task as a sequential Bayesian experimental design problem, where the log-likelihood function is modelled with a hierarchical Gaussian process (GP) surrogate model, which is used to efficiently select additional log-likelihood evaluation locations. Motivated by recent progress in the related problem of batch Bayesian optimisation, we develop various batch-sequential design strategies which allow to run some of the potentially costly simulations in parallel. We analyse the properties of the resulting method theoretically and empirically. Experiments with several toy problems and simulation models suggest that our method is robust, highly parallelisable, and sample-efficient.

Nesterov-aided stochastic gradient methods using Laplace approximation for Bayesian design optimization

Finding the best setup for experiments is the primary concern for Optimal Experimental Design (OED). Here, we focus on the Bayesian experimental design problem of finding the setup that maximizes the Shannon expected information gain. We use the stochastic gradient descent and its accelerated counterpart, which employs Nesterov’s method, to solve the optimization problem in OED. We adapt a restart technique, originally proposed for the acceleration in deterministic optimization, to improve stochastic optimization methods. We combine these optimization methods with three estimators of the objective function: the double-loop Monte Carlo estimator (DLMC), the Monte Carlo estimator using the Laplace approximation for the posterior distribution (MCLA) and the double-loop Monte Carlo estimator with Laplace-based importance sampling (DLMCIS). Using stochastic gradient methods and Laplace-based estimators together allows us to use expensive and complex models, such as those that require solving partial differential equations (PDEs). From a theoretical viewpoint, we derive an explicit formula to compute the gradient estimator of the Monte Carlo methods, including MCLA and DLMCIS. From a computational standpoint, we study four examples: three based on analytical functions and one using the finite element method. The last example is an electrical impedance tomography experiment based on the complete electrode model. In these examples, the accelerated stochastic gradient descent method using MCLA converges to local maxima with up to five orders of magnitude fewer model evaluations than gradient descent with DLMC.

Near-optimal discrete optimization for experimental design: a regret minimization approach

The experimental design problem concerns the selection of k points from a potentially large design pool of p-dimensional vectors, so as to maximize the statistical efficiency regressed on the selected k design points. Statistical efficiency is measured by optimality criteria, including A(verage), D(eterminant), T(race), E(igen), V(ariance) and G-optimality. Except for the T-optimality, exact optimization is NP-hard. We propose a polynomial-time regret minimization framework to achieve a $(1+\varepsilon)$ approximation with only $O(p/\varepsilon^2)$ design points, for all the optimality criteria above. In contrast, to the best of our knowledge, before our work, no polynomial-time algorithm achieves $(1+\varepsilon)$ approximations for D/E/G-optimality, and the best poly-time algorithm achieving $(1+\varepsilon)$-approximation for A/V-optimality requires $k = \Omega(p^2/\varepsilon)$ design points.

Effective Parameterization of PEM Fuel Cell Models—Part II: Robust Parameter Subset Selection, Robust Optimal Experimental Design, and Multi-Step Parameter Identification Algorithm

The second part of this two-part study develops a systematic framework for parameter identification in polymer electrolyte membrane (PEM) fuel cell models. The framework utilizes the extended local sensitivity results of the first part to find an optimal subset of parameters for identification. This is achieved through an optimization algorithm that maximizes the well-known D-optimality criterion. The sensitivity data are then used for optimal experimental design (OED) to ensure that the resulting experiments are maximally informative for the purpose of parameter identification. To make the experimental design problem computationally tractable, the optimal experiments are chosen from a predefined library of operating conditions. Finally, a multi-step identification algorithm is proposed to formulate a regularized and well-conditioned optimization problem. The identification algorithm utilizes the unique structure of output predictions, wherein sensitivities to parameter perturbations typically vary with the load. To verify each component of the framework, synthetic experimental data generated with the model using nominal parameter values are used in an identification case study. The results confirm that each of these components plays a critical role in successful parameter identification.

A Scheme for Simultaneous Optimal Tracking Control and Experiment Design

This paper is about the formulation and numerical solutions of simultaneous optimal experiment design and optimal tracking control problems. Our motivating example is a robot arm that is mounted on a kitchen wall over the hotplates in order to assist humans with cooking. The robot arm can take over simple tasks such as stirring, automatic seasoning, or adding ingredients to a pot following a given recipe. Here, one of the main challenges is that the robot has to learn about the mass, inertial and other properties of the objects it is picking up while satisfying control tasks. Thus, we are facing a classical dual control problem, where system excitation for the purpose of learning has to be trade-off with other objectives such as tracking performance. After reviewing existing approaches, we propose a new formulation which allows to implement a trade-off between experiment design as well as tracking objectives in a systematic way by exploiting recent ideas from the field of economic experiment design. The approach is tested with a dynamic robot arm model performing a simple but illustrative cooking maneuver where learning and control goals are in conflict.

Design and Analysis of Switchback Experiments

Switchback experiments, where a firm sequentially exposes an experimental unit to random treatments, are among the most prevalent designs used in the technology sector, with applications ranging from ride-hailing platforms to online marketplaces. Although practitioners have widely adopted this technique, the derivation of the optimal design has been elusive, hindering practitioners from drawing valid causal conclusions with enough statistical power. We address this limitation by deriving the optimal design of switchback experiments under a range of different assumptions on the order of the carryover effect --- the length of time a treatment persists in impacting the outcome. We cast the optimal experimental design problem as a minimax discrete optimization problem, identify the worst-case adversarial strategy, establish structural results, and solve the reduced problem via a continuous relaxation. For switchback experiments conducted under the optimal design, we provide two approaches for performing inference. The first provides exact randomization based p-values, and the second uses a new finite population central limit theorem to conduct conservative hypothesis tests and build confidence intervals. We further provide theoretical results when the order of the carryover effect is misspecified and provide a data-driven procedure to identify the order of the carryover effect. We conduct extensive simulations to study the empirical properties of our results and conclude with practical suggestions.