Sequential Experimental Design Research Articles

Strategies for sequential design of experiments and augmentation

AbstractThe benefits of sequential design of experiments have long been described for both model‐based and space‐filling designs. However, in our experience, too few practitioners take advantage of the opportunity afforded by this approach to maximize the learning from their experimentation. By obtaining data sequentially, it is possible to learn from the early stages to inform subsequent data collection, minimize wasted resources, and provide answers for a series of objectives for the overall experiment. This paper provides methods and algorithms to create augmented distance‐based space‐filling designs, using both uniform and non‐uniform space‐filling strategies, that can be constructed at each stage based on information learned in earlier stages. We illustrate the methods with several examples that involve different initial data, types of space‐filing designs and experimental goals.

Multivariate optimization of low-temperature cleanup followed by dispersive solid-phase extraction for detection of Bisphenol A and benzophenones in infant formula

A simple and effective analytical method to determine six contaminants, including five benzophenones (BP, BP-1, BP-3, BP-8, and BP-12) and bisphenol A (BPA) in infant formulas was developed in this study. For this, a sequential experimental design was used to optimize the extraction and cleanup method using low temperature partition (LTP) combined with dispersive solid phase extraction (dSPE). The effect of primary secondary amine (PSA), sodium chloride (NaCl), graphitized carbon black (GCB), octadecyl (C18), strong anion exchanger (SAX), water, acetonitrile (ACN) and, ultrasound (US) time were evaluated using a sequential design of experiments including a Plackett-Burman, a central composite rotatable design, and the Derringer and Suich's tool. The method was validated, and it showed a limit of quantification varying from 0.06 to 2 mg.kg-1, good precision (< 20% RSD), and recovery (52-106%). The method proposed was applied to twenty-five samples of commercial infant formulas.

A model-based framework assisting the design of vapor-liquid equilibrium experimental plans

In this paper we propose a framework for Model-based Sequential Optimal Design of Experiments to assist experimenters involved in Vapor-Liquid equilibrium characterization studies to systematically construct thermodynamically consistent models. The approach uses an initial continuous optimal design obtained via semidefinite programming, and then iterates between two stages (i) model fitting using the information available; and (ii) identification of the next experiment, so that the information content in data is maximized. The procedure stops when the number of experiments reaches the maximum for the experimental program or the dissimilarity between the parameter estimates during two consecutive iterations is below a given threshold. This methodology is exemplified with the D-optimal design of isobaric experiments, for characterizing binary mixtures using the NRTL and UNIQUAC thermodynamic models for liquid phase. Significant reductions of the confidence regions for the parameters are achieved compared with experimental plans where the observations are uniformly distributed over the domain.

Sequential Bayesian experiment design for optically detected magnetic resonance of nitrogen-vacancy centers.

In magnetometry using optically detected magnetic resonance of nitrogen vacancy (NV-) centers, we demonstrate more than one order-of-magnitude speed up with sequential Bayesian experiment design as compared with conventional frequency-swept measurements. The NV- center is an excellent platform for magnetometry with potential spatial resolution down to few nanometers and demonstrated single-defect sensitivity down to nT/Hz1/2. The NV- center is a quantum defect with spin S = 1 and coherence time up to several milliseconds at room temperature. Zeeman splitting of the NV- energy levels allows detection of the magnetic field via photoluminescence. We compare conventional NV- center photoluminescence measurements that use pre-determined sweeps of the microwave frequency with measurements using a Bayesian inference methodology. In sequential Bayesian experiment design, the settings with maximum utility are chosen for each measurement in real time based on the accumulated experimental data. Using this method, we observe an order of magnitude decrease in the NV- magnetometry measurement time necessary to achieve a set precision.

Sequential Online Dispatch in Design of Experiments for Single- and Multiple-Response Surrogate Modeling

As parallel computing becomes increasingly important in many real-world applications, a batch sequential experimental design (BSED), which adds a batch of computer experiments per iteration and runs these simulations in parallel, is gaining popularity in surrogate modeling. This article proposes sequential online dispatch in design of experiments (SODDE) for single- and multiple-response surrogate modeling when multiple processors work in parallel but not independently. The proposed method includes several unique features: 1) it works with any popular acquisition function to select a single new sample point at each iteration; 2) it minimizes the idle time of all processors; 3) it rapidly updates the surrogate model; and 4) it dynamically reconstructs the surrogate model when a simulation process aborts, minimizing the impact incurred by the abortion. The effectiveness of SODDE is evaluated in one mathematical example and one industrial problem. The latter problem considers blade design of horizontal axis wind turbines (HAWTs). The cost of finding blade geometry that results in the desired aerodynamic behavior of HAWTs is estimated for BSED and SODDE. Relative to BSED, SODDE reduces the costs by up to approximately 31%. Note to Practitioners —Computer simulation models are often used to represent the behavior of complex engineered systems, such as automobiles, aircraft, and wind turbines. Although running computer simulations allows analyzing the performance of these systems, the simulations can still be computationally costly (e.g., hours of runtime for numerically solving a large number of differential equations in one simulation), which makes the performance analysis task challenging. It is important to design an innovative simulation process that replaces the original expensive simulation models by cheap-to-build surrogate models, and at the same time, minimize the computational costs for building the surrogate models. The new method developed in this article allows experimental designers to run batch sequential iterations and run these iterations in parallel. This, in engineering practice, will help meet the need for long and complex experimental designs scenarios in a parallel computing environment, where the computer “up time” (or runtime) needs to be maximized to save time and ultimately, money. Cost analysis results from an industrial problem involving the inverse design of wind turbine blades suggest that the proposed method reduces the computational costs by up to approximately 31% over an existing method.

Editorial for the special issue on advances in forward and inverse surrogate modeling for high‐frequency design

Editorial for the special issue on advances in forward and inverse surrogate modeling for <scp>high‐frequency</scp> design

Retraction of “Optimization of a Carbon Dioxide-Assisted Nanoparticle Deposition Process Using Sequential Experimental Design with Adaptive Design Space”

RETURN TO ISSUEPREVRetractionNEXTORIGINAL ARTICLEThis notice is a retraction.Retraction of “Optimization of a Carbon Dioxide-Assisted Nanoparticle Deposition Process Using Sequential Experimental Design with Adaptive Design Space”Michael J. Casciato*Michael J. CasciatoMore by Michael J. Casciato, Sungil KimSungil KimMore by Sungil Kim, J. C. LuJ. C. LuMore by J. C. Lu, Dennis W. HessDennis W. HessMore by Dennis W. Hess, and Martha A. GroverMartha A. GroverMore by Martha A. Groverhttp://orcid.org/0000-0002-7036-776XCite this: Ind. Eng. Chem. Res. 2020, 59, 35, 15807Publication Date (Web):August 19, 2020Publication History Published online19 August 2020Published inissue 2 September 2020https://doi.org/10.1021/acs.iecr.0c03481Copyright © 2020 American Chemical SocietyRIGHTS & PERMISSIONSArticle Views550Altmetric-Citations-LEARN ABOUT THESE METRICSArticle Views are the COUNTER-compliant sum of full text article downloads since November 2008 (both PDF and HTML) across all institutions and individuals. These metrics are regularly updated to reflect usage leading up to the last few days.Citations are the number of other articles citing this article, calculated by Crossref and updated daily. Find more information about Crossref citation counts.The Altmetric Attention Score is a quantitative measure of the attention that a research article has received online. Clicking on the donut icon will load a page at altmetric.com with additional details about the score and the social media presence for the given article. Find more information on the Altmetric Attention Score and how the score is calculated. Share Add toView InAdd Full Text with ReferenceAdd Description ExportRISCitationCitation and abstractCitation and referencesMore Options Share onFacebookTwitterWechatLinked InReddit PDF (383 KB) Get e-Alerts Get e-Alerts

Exploratory-Phase-Free Estimation of GP Hyperparameters in Sequential Design Methods-At the Example of Bayesian Inverse Problems.

Methods for sequential design of computer experiments typically consist of two phases. In the first phase, the exploratory phase, a space-filling initial design is used to estimate hyperparameters of a Gaussian process emulator (GPE) and to provide some initial global exploration of the model function. In the second phase, more design points are added one by one to improve the GPE and to solve the actual problem at hand (e.g., Bayesian optimization, estimation of failure probabilities, solving Bayesian inverse problems). In this article, we investigate whether hyperparameters can be estimated without a separate exploratory phase. Such an approach will leave hyperparameters uncertain in the first iterations, so the acquisition function (which tells where to evaluate the model function next) and the GPE-based estimator need to be adapted to non-Gaussian random fields. Numerical experiments are performed exemplarily on a sequential method for solving Bayesian inverse problems. These experiments show that hyperparameters can indeed be estimated without an exploratory phase and the resulting method works almost as efficient as if the hyperparameters had been known beforehand. This means that the estimation of hyperparameters should not be the reason for including an exploratory phase. Furthermore, we show numerical examples, where these results allow us to eliminate the exploratory phase to make the sequential design method both faster (requiring fewer model evaluations) and easier to use (requiring fewer choices by the user).

Sequential Bayesian Experimental Design for Implicit Models via Mutual Information

Bayesian experimental design (BED) is a framework that uses statistical models and decision making under uncertainty to optimise the cost and performance of a scientific experiment. Sequential BED, as opposed to static BED, considers the scenario where we can sequentially update our beliefs about the model parameters through data gathered in the experiment. A class of models of particular interest for the natural and medical sciences are implicit models, where the data generating distribution is intractable, but sampling from it is possible. Even though there has been a lot of work on static BED for implicit models in the past few years, the notoriously difficult problem of sequential BED for implicit models has barely been touched upon. We address this gap in the literature by devising a novel sequential design framework for parameter estimation that uses the Mutual Information (MI) between model parameters and simulated data as a utility function to find optimal experimental designs, which has not been done before for implicit models. Our approach uses likelihood-free inference by ratio estimation to simultaneously estimate posterior distributions and the MI. During the sequential BED procedure we utilise Bayesian optimisation to help us optimise the MI utility. We find that our framework is efficient for the various implicit models tested, yielding accurate parameter estimates after only a few iterations.

Robust design methodology for single phase induction motor ceiling fan

This paper deals with the robust design of a single-phase induction motor (SPIM) for ceiling fan application. It also identifies the critical design parameters of a single-phase induction motor for an outer rotor geometry. The inaccuracy issues in the equivalent circuit method for such low watt application, are discussed with experimental validation using different equivalent circuit approaches. Moreover, the finite element (Finite Element) approach is used for the accurate design of the SPIMs. The magnetic behaviour of the ceiling fan SPIM is also studied in detail. A 45 W ceiling fan design is considered for improvement in the efficiency and performance with the same outer and inner diameter dimensions. For the improved design of the SPIM, a sequential DoE (Design of Experiments) based methodology is used. Taguchi's orthogonal arrays, namely L18 and L9, are used for robust design of the stator as well as the rotor. The improved design for ceiling fan SPIM is validated by two-dimensional time stepping transient FEA.

Sequential model-based design of experiments for development of mathematical models for thin film deposition using chemical vapor deposition process

Experimental data is required to estimate unknown parameters for a mathematical model describing thin film growth and to optimize the chemical vapor deposition (CVD) process. The most informative experimental data can be obtained by sequential model-based design of experiments (MBDoE), but this requires significant effort to implement compared to factorial design of experiments (DoE). For CVD processes it is not clear which approach is better or more efficient. Here, we compare the effectiveness of sequential A-optimal (MBDoE) with a factorial (DoE) using a case study involving a mechanistic model of thin-film deposition. The sequential A-optimal MBDoE and factorial DoE are compared based on their effectiveness in obtaining accurate parameter estimates and model predictions. Synthetic experimental data is generated from simulations with the true parameters plus noise. The results suggest that the A-optimal MBDoE is slightly better than factorial DoE for estimating model parameters. The parameter estimates from each experimental design approach are used to solve a model-based optimization problem. The results indicate that model-based optimization from the model with parameter estimates obtained by the A-optimal design gives a significantly better estimate of the optimal process conditions than factorial DoE.

Sequential experimental design for predator-prey functional response experiments.

Understanding functional response within a predator-prey dynamic is a cornerstone for many quantitative ecological studies. Over the past 60 years, the methodology for modelling functional response has gradually transitioned from the classic mechanistic models to more statistically oriented models. To obtain inferences on these statistical models, a substantial number of experiments need to be conducted. The obvious disadvantages of collecting this volume of data include cost, time and the sacrificing of animals. Therefore, optimally designed experiments are useful as they may reduce the total number of experimental runs required to attain the same statistical results. In this paper, we develop the first sequential experimental design method for predator-prey functional response experiments. To make inferences on the parameters in each of the statistical models we consider, we use sequential Monte Carlo, which is computationally efficient and facilitates convenient estimation of important utility functions. It provides coverage of experimental goals including parameter estimation, model discrimination as well as a combination of these. The results of our simulation study illustrate that for predator-prey functional response experiments sequential design outperforms static design for our experimental goals. R code for implementing the methodology is available via https://github.com/haydenmoffat/sequential_design_for_predator_prey_experiments.

Rainbow in the dark. The identification of diagnostic projectile impact features on rock crystal

Over the last years, functional analyses and projectile fracture analyses on non-chert tools are becoming widespread and are drawing more attention in the archaeological literature. The actual tendencies of conducting holistic studies of the archaeological sites and their materials is allowing to obtain a better quality of data and a better understanding of the subsistence patterns of hunter-gatherer societies. In this way, rock crystal and other types of minerals and rocks of the quartzose group are getting a significant presence in the literature. In this paper, we intend to address a comprehensive approach, both macroscopic and microscopic, to an assemblage of experimental rock crystal projectiles to shed light on the identification of the diagnostic features of the use of small rock crystal pieces as projectile points. Aiming to avoid misidentifications, we did not only follow a sequential experiment design, but also carried out a comparative analysis between impact fractures and knapping fractures, marks from bipolar knapping on an anvil, and intentional bending fractures. Besides, in the elaboration of the experimental projectile assemblage we used different sets of targets to document possible differences in the quantity and disposition of the wear impact traces.

Parallel Gaussian Process Surrogate Bayesian Inference with Noisy Likelihood Evaluations

We consider Bayesian inference when only a limited number of noisy log-likelihood evaluations can be obtained. This occurs for example when complex simulator-based statistical models are fitted to data, and synthetic likelihood (SL) method is used to form the noisy log-likelihood estimates using computationally costly forward simulations. We frame the inference task as a sequential Bayesian experimental design problem, where the log-likelihood function is modelled with a hierarchical Gaussian process (GP) surrogate model, which is used to efficiently select additional log-likelihood evaluation locations. Motivated by recent progress in the related problem of batch Bayesian optimisation, we develop various batch-sequential design strategies which allow to run some of the potentially costly simulations in parallel. We analyse the properties of the resulting method theoretically and empirically. Experiments with several toy problems and simulation models suggest that our method is robust, highly parallelisable, and sample-efficient.

An adaptive sequential experiment design method for model validation

Efficient experiment design is of great significance for the validation of simulation model with high nonlinearity and large input space. Excessive validation experiment raises the cost while insufficient test increases the risks of accepting an invalid model. In this paper, an adaptive sequential experiment design method combining global exploration criterion and local exploitation criterion is proposed. The exploration criterion utilizes discrepancy metric to improve the space-filling property of the design points while the exploitation criterion employs the leave one out error to discover informative points. To avoid the clustering of samples in the local region, an adaptive weight updating approach is provided to maintain the balance between exploration and exploitation. Besides, the credibility distribution function characterizing the relationship between the input and result credibility is introduced to support the model validation experiment design. Finally, six benchmark problems and an engineering case are applied to examine the performance of the proposed method. The experiments indicate that the proposed method achieves satisfactory performance for function approximation in accuracy and convergence.

Adaptive design of experiments for global Kriging metamodeling through cross-validation information

This paper discusses a new sequential adaptive design of experiments (DoE) approach for global Kriging metamodeling applications. The sequential implementation is established by using the current metamodel, formulated based on the existing experiments, to guide the selection of the optimal new experiment(s). The score function, defining the DoE objective, combines two components: (1) the metamodel prediction variability, expressed through the predictive variance, and (2) the metamodel bias, approximated through the leave-one-out cross validation (LOOCV) error. The latter is used as a weighting factor to extend traditional DoE approaches that focus solely on the metamodel prediction variability. Two such approaches are considered here, adopting either the integrated mean squared error or the maximum mean squared error as the basic component of the score function. The incorporation of bias information as weighting within these well-established approaches facilitates a direct extension of their respective computational workflows, making the proposed implementation attractive from computational perspective. An efficient optimization scheme for identification of the next experiment, as well as the balancing of exploration and exploitation between the two components of the score function, are also discussed. The incorporation of LOOCV weightings is shown to be highly beneficial in a total of six analytical and engineering examples. Furthermore, these examples demonstrate that for DoE approaches which use LOOCV information as weights, it is preferable to update the predictive variance to explicitly consider the impact of the new experiment, rather than relying strictly on the current metamodel variance.

Robust adaptive least squares polynomial chaos expansions in high‐frequency applications

AbstractWe present an algorithm for computing sparse, least squares‐based polynomial chaos expansions, incorporating both adaptive polynomial bases and sequential experimental designs. The algorithm is employed to approximate stochastic high‐frequency electromagnetic models in a black‐box way, in particular, given only a dataset of random parameter realizations and the corresponding observations regarding a quantity of interest, typically a scattering parameter. The construction of the polynomial basis is based on a greedy, adaptive, sensitivity‐related method. The sequential expansion of the experimental design employs different optimality criteria, with respect to the algebraic form of the least squares problem. We investigate how different conditions affect the robustness of the derived surrogate models, that is, how much the approximation accuracy varies given different experimental designs. It is found that relatively optimistic criteria perform on average better than stricter ones, yielding superior approximation accuracies for equal dataset sizes. However, the results of strict criteria are significantly more robust, as reduced variations regarding the approximation accuracy are obtained, over a range of experimental designs. Two criteria are proposed for a good accuracy‐robustness trade‐off.

An Improved Structural Reliability Analysis Method Based on Local Approximation and Parallelization

The Kriging-based reliability method with a sequential design of experiments (DoE) has been developed in recent years for implicit limit state functions. Such methods include the efficient global reliability analysis, the active learning reliability method combining Kriging and MCS Simulations. In this research, a novel local approximation method based on the most probable failure point (MPFP) is proposed to improve such methods. In this method, the MPFP calculated in the last iteration is the center of the next sampling region. The size of the local region depends on the reliability index obtained by the First Order Reliability Method (FORM) and the deviation distance of the standard deviation. The proposed algorithm, which approximates the limit state function accurately near MPFP rather than in the whole design space, can avoid selecting samples in regions that have negligible effects on the reliability analysis results. In addition, a multi-point enrichment technique is also introduced to select multiple sample points in each iteration. After the high-quality approximation of limit state function is obtained, the failure probability is calculated by the Monte Carlo method. Four numerical examples are used to validate the accuracy and efficiency of the proposed method. Results show that the proposed method is very effective for an accurate evaluation of the failure probability.

Holistic development of coal tar lotion by embedding design of experiments (DoE) technique: preclinical investigations

ABSTRACTBackground: The research work endeavors to develop a liquid dosage form of an efficacious antipsoriatic drug, i.e., coal tar, but having problems like variability and patient noncompliance.Methods: The emulsion was prepared by the wet gum method from standardized coal tar. The optimized lotion obtained after sequential experimental designs was characterized for various dosage form and/or coal tar-related properties including efficacy.Results: The formulation deposited more coal tar in the unit area of rat skin than marketed lotions. The efficacy of lotion in psoriasis animal models was more or equivalent to marketed lotions. The formulation showed one compartment body model dermatokinetics, nonirritancy after repeated applications, and stability at room conditions for a year.Conclusion: The formulation with desired attributes was successfully developed.

Development of a framework for sequential Bayesian design of experiments: Application to a pilot-scale solvent-based CO2 capture process

In this paper, a methodology is developed for sequential design of experiments (SDoE) for process systems and applied to a solvent-based CO2 capture system. In this approach, the prior knowledge of the system is used to prioritize process data collection at specific operating conditions. These data are then incorporated into a Bayesian inference methodology for updating a stochastic model by refining estimations of its underlying parameters, and the updated model is then used to generate the next set of test runs. Thus, the new knowledge obtained from the data is used to guide subsequent iterations of the experimental runs, ensuring that the overall data collection is maximally informative given that most experimental campaigns, especially at pilot or higher-scale plants, are costly, time-consuming, and resource-limited. The test run objective for this work was to minimize the maximum model prediction uncertainty for key output variables, but the methodology is generic and can be readily applied to other test run objectives. This methodology is applied to an aqueous monoethanolamine (MEA) pilot plant campaign at the National Carbon Capture Center (NCCC) in Wilsonville, Alabama, USA. The SDoE framework was utilized for two iterations, while collecting 18 sets of data representing different process conditions, and this resulted in an overall average reduction in uncertainty of approximately 50% in the prediction of CO2 capture percentage. Moreover, 11 additional data sets were obtained with variation of absorber packing height for further model validation. This work shows the capability of the SDoE framework to maximize learning given limited resources, allowing for the reduction of model uncertainty, which is of great importance for many applications including reduction of technical risk associated with scale-up and economic analysis.