Adaptive Experimental Design Research Articles

Accelerating development in wood‐plastic composites: Presenting key material properties and demonstrating statistical methods

AbstractIn a time when global initiatives to lower CO2 emissions are accelerating the shift towards bio‐based products, various efforts are being made to develop wood plastic composites (WPCs) and similar bio‐composites using a variety of raw materials. This paper first showcases benchmark material properties serving as standards and models the composition, for polypropylene‐based WPCs integrating wood flour and maleic anhydride‐modified polypropylene. Subsequently, by applying statistical modeling techniques, we demonstrate an adaptive experimental design that efficiently and rapidly adjusts formulations. Beginning with an empirical dataset (24 samples), we established high‐performing models that articulate the individual contributions of the constituents on ten vital properties of WPCs. Our adaptive experimental design, leveraging nonlinear partial least squares regression and the Bayesian optimization, successfully refined formulations to enhance bending and impact strengths. Notably, we proposed optimal conditions starting from just eight samples with minimal iterations. This study shows that statistical techniques can quickly optimize WPC formulations, making the overall development process faster. By addressing the multiple conflicting properties, these techniques can greatly reduce the time and effort needed for development.Highlights Applied several regression methods to analyze ten properties of PP‐based WPCs. Showcased distinct impacts of the formulations of WF and MAPP on WPCs. Implemented the adaptive experimental design for formulation optimization. Enhanced bending and impact strengths in the formulations of WPCs. Advanced sustainable material development with data‐driven approach.

Adaptive design of experiments to fit surrogate Gaussian process regression models allows fast sensitivity analysis of the input waveform for patient-specific 3D CFD models of liver radioembolization

Background and objectivePatient-specific 3D computational fluid dynamics (CFD) models are increasingly being used to understand and predict transarterial radioembolization procedures used for hepatocellular carcinoma treatment. While sensitivity analyses of these CFD models can help to determine the most impactful input parameters, such analyses are computationally costly. Therefore, we aim to use surrogate modelling to allow relatively cheap sensitivity analysis. As an example, we compute Sobol's sensitivity indices for three input waveform shape parameters. MethodsWe extracted three characteristic shape parameters from our input mass flow rate waveform (peak systolic mass flow rate, heart rate, systolic duration) and defined our 3D input parameter space by varying these parameters within 75 %-125 % of their nominal values. To fit our surrogate model with a minimal number of costly CFD simulations, we developed an adaptive design of experiments (ADOE) algorithm. The ADOE uses 100 Latin hypercube sampled points in 3D input space to define the initial design of experiments (DOE). Subsequently, we re-sample input space with 10,000 Latin Hypercube sampled points and cheaply estimate the outputs using the surrogate model. In each of 27 equivolume bins which divide our input space, we determine the most uncertain prediction of the 10,000 points, compute the true outputs using CFD, and add these points to the DOE. For each ADOE iteration, we calculate Sobol's sensitivity indices, and we continue to add batches of 27 samples to the DOE until the Sobol indices have stabilized. ResultsWe tested our ADOE algorithm on the Ishigami function and showed that we can reliably obtain Sobol's indices with an absolute error <0.1. Applying ADOE to our waveform sensitivity problem, we found that the first-order sensitivity indices were 0.0550, 0.0191 and 0.407 for the peak systolic mass flow rate, heart rate, and the systolic duration, respectively. ConclusionsAlthough the current study was an illustrative case, the ADOE allows reliable sensitivity analysis with a limited number of complex model evaluations, and performs well even when the optimal DOE size is a priori unknown. This enables us to identify the highest-impact input parameters of our model, and other novel, costly models in the future.

Adaptive Design of Experiments for Fault Injection Testing of Highly Automated Vehicles

Adaptive Design of Experiments for Fault Injection Testing of Highly Automated Vehicles

Parallelization of adaptive Bayesian cubature using multimodal optimization algorithms

PurposeBayesian cubature (BC) has emerged to be one of most competitive approach for estimating the multi-dimensional integral especially when the integrand is expensive to evaluate, and alternative acquisition functions, such as the Posterior Variance Contribution (PVC) function, have been developed for adaptive experiment design of the integration points. However, those sequential design strategies also prevent BC from being implemented in a parallel scheme. Therefore, this paper aims at developing a parallelized adaptive BC method to further improve the computational efficiency.Design/methodology/approachBy theoretically examining the multimodal behavior of the PVC function, it is concluded that the multiple local maxima all have important contribution to the integration accuracy as can be selected as design points, providing a practical way for parallelization of the adaptive BC. Inspired by the above finding, four multimodal optimization algorithms, including one newly developed in this work, are then introduced for finding multiple local maxima of the PVC function in one run, and further for parallel implementation of the adaptive BC.FindingsThe superiority of the parallel schemes and the performance of the four multimodal optimization algorithms are then demonstrated and compared with the k-means clustering method by using two numerical benchmarks and two engineering examples.Originality/valueMultimodal behavior of acquisition function for BC is comprehensively investigated. All the local maxima of the acquisition function contribute to adaptive BC accuracy. Parallelization of adaptive BC is realized with four multimodal optimization methods.

(Non-)Parametric Recoverability of Preferences and Choice Prediction

Abstract Simple functional forms for utility require restrictive structural assumptions that are often contrary to observed behavior. Even so, they are widely used in applied economic research. I address this issue using a two-part adaptive experimental design to compare the predictions of a popular parametric model of decision making under risk to those of non-parametric bounds on indifference curves. Interpreting the latter as an approximate upper bound, I find the parametric model sacrifices very little in terms of predictive success. This suggests that, despite their restrictiveness, simple functional forms may nevertheless be useful representations of preferences over risky alternatives.

Traditional or adaptive design of experiments? A pilot-scale comparison on wood delignification

Traditional design of experiments and response surface methodology are widely used in engineering and process development. Bayesian optimization is an alternative machine learning approach that adaptively selects successive experimental conditions based on a predefined performance measure. Here we compared the two approaches using simulations and empirical experiments on alkaline wood delignification to identify important benefits and drawbacks of Bayesian optimization in the context of design of experiments. The simulations showed that the selection of initial experiments and measurement noise influenced the convergence of the Bayesian optimization algorithm to known optimal conditions. Both methods, however, showed comparable pilot-scale results on optimal digestion conditions, where high cellulose yields were combined with acceptable kappa numbers and pulp viscosities. Bayesian optimization did not enable a decrease in the number of experiments required for reaching these conditions but provided a more accurate model in the vicinity of the optimum based on additional modelling and cross-validation. These results shed light on the practical differences between the two methodologies for process development and are an important contribution to the chemometrics and machine learning communities.

Adaptive design of experiments guided by an active learning approach

Fault detection in a manufacturing process is often challenging due to a lack of system background information. Design of Experiments (DoE) are used for effective planning of experiments to get knowledge of the unknown system. Those DoE lead to many experiments to depict the complex relationships of the reasons and effects. The challenge is the optimization to reduce the number of experiments while maintaining accuracy. This paper presents a novel approach for a guided DoE based on a Deep Active Learning (DeepAL) strategy to drastically downsize the number of experiments in order to avoid high execution costs. The DeepAL uses a new approach in uncertainty rating of the experiment space by using diversity rating to improve a faster generalization of the system approximation. Empirical evaluations show on average 60% better performance of the novel approach in combination with Bayesian neural networks compared to other methods.

Active Adaptive Experimental Design for Treatment Effect Estimation with Covariate Choices

Active Adaptive Experimental Design for Treatment Effect Estimation with Covariate Choices

Machine Learning for Biological Design.

We briefly present machine learning approaches for designing better biological experiments. These approaches build on machine learning predictors and provide additional tools to guide scientific discovery. There are two different kinds of objectives when designing better experiments: to improve the predictive model or to improve the experimental outcome. We survey five different approaches for adaptive experimental design that iteratively search the space of possible experiments while adapting to measured data. The approaches are Bayesian optimization, bandits, reinforcement learning, optimal experimental design, and active learning. These machine learning approaches have shown promise in various areas of biology, and we provide broad guidelines to the practitioner and links to further resources.

Optimal Experimental Design for Staggered Rollouts

In this paper, we study the design and analysis of experiments conducted on a set of units over multiple time periods in which the starting time of the treatment may vary by unit. The design problem involves selecting an initial treatment time for each unit in order to most precisely estimate both the instantaneous and cumulative effects of the treatment. We first consider nonadaptive experiments, in which all treatment assignment decisions are made prior to the start of the experiment. For this case, we show that the optimization problem is generally NP-hard, and we propose a near-optimal solution. Under this solution, the fraction entering treatment each period is initially low, then high, and finally low again. Next, we study an adaptive experimental design problem, in which both the decision to continue the experiment and treatment assignment decisions are updated after each period’s data are collected. For the adaptive case, we propose a new algorithm, the precision-guided adaptive experiment algorithm, which addresses the challenges at both the design stage and the stage of estimating treatment effects, ensuring valid post-experiment inference, accounting for the adaptive nature of the design. Using realistic settings, we demonstrate that our proposed solutions can reduce the opportunity cost of the experiments by more than 50%, compared with static design benchmarks. This paper was accepted by George Shanthikumar, data science. Funding: S. Athey and G. Imbens were supported by the Office of Naval Research [Grant N00014-19-1-2468]. M. Bayati was supported by the National Science Foundation [Grant CMMI 1554140]. Supplemental Material: The e-companion and data are available at https://doi.org/10.1287/mnsc.2023.4928 .

Robust Design of a Dimethyl Ether Production Process Using Process Simulation and Robust Bayesian Optimization.

As greenhouse gases such as CO2 continue to promote global warming, the reduction of CO2 emissions is attracting increasing attention. In this study, we design a process for producing dimethyl ether (DME), which is a promising means of using CO2 as a resource. Design variables such as temperature and pressure need to be optimized to reduce CO2 emissions while maintaining high product purity and DME production. Conventional process designs determine these design variables from the chemical background and through trial-and-error simulations, which are very time-consuming. The proposed method optimizes the design variables efficiently by repeating the process simulations and selecting promising candidates for the design variables using machine learning. For an adaptive design of experiments, Bayesian optimization is used to achieve the objectives of the DME process while efficiently optimizing the design variables. In addition, we also optimize the design variables considering variations in the temperature and pressure data, meaning robust Bayesian optimization. The proposed method successfully identifies design variables that satisfy all experimental targets in an average of 54 simulations while achieving 100% of the targets with product purity 0.95-1.00, amount of DME in the product 350-845 kmol/h, and CO2 emissions 0-835 kmol/h, confirming the effectiveness of the proposed robust Bayesian optimization method.

Computationally Efficient Adaptive Design of Experiments for Global Metamodeling through Integrated Error Approximation and Multicriteria Search Strategies

Gaussian processes (GPs) are a popular technique for global metamodeling applications. Their objective in such settings is to establish an efficient and globally accurate approximation of the response surface of computationally expensive simulation models. When developing such GPs, the design of (simulation) experiments (DoE) plays an important role in reducing the required number of model runs for obtaining accurate approximations. Sequential (adaptive) selection of experiments can provide significant advantages, especially when the response surface is characterized by localized nonlinearities. Such adaptive DoE strategies for global metamodeling applications typically focus on minimizing the predictive GP variance, representing an exploration strategy, while recent developments have additionally considered the reduction of the GP bias obtained through cross validation, representing an exploitation strategy. While significant focus has been placed on the definition of appropriate adaptive DoE criteria, computational challenges still exist that limit the widespread adoption of adaptive DoE techniques—for example, related to the additional computational demand for identifying the optimal new experiment(s) or the necessity to establish proper schemes to combine exploration and exploitation strategies. To address these specific challenges, this research investigates two new adaptive DoE formulations. The first one focuses on the approximation of the popular integrated mean square error (IMSE) DoE criterion. The computationally demanding GP predictive variance update (after addition of each candidate experiment), required in the original IMSE formulation, is replaced by an approximation based on the current predictive variance and the domain of influence that surrounds each new experiment. The approximation is established through a parametric formulation that leverages the GP kernel to describe the aforementioned domain, with characteristics that are progressively calibrated across the GP training stages, to minimize the discrepancy between the actual and the approximated IMSE. The second formulation establishes a multicriteria search for simultaneously identifying multiple Pareto optimal experiments that balance exploration and exploitation objectives, replacing conventional strategies that establish a weighted combination of these objectives to promote a single DoE selection criterion.

Global Model Calibration of High-Temperature Gas-Cooled Reactor Pebble-Bed Module Using an Adaptive Experimental Design

The world’s first high-temperature gas-cooled reactor pebble-bed module (HTR-PM) nuclear power plant adopts an innovative reactor type and a modular structure design. Parameter estimation and model calibration are of great significance prior to the implementation of model-based control and optimization. This paper focuses on identifying the thermal hydraulic parameters of HTR-PM over the global operating domain. The process technology and model mechanism of HTR-PM are reviewed. A parameter submodel named global parameter mapping is presented to quantify the relationship between an unknown model parameter and different operating conditions in a data-driven manner. The ideal construction of such a mapping requires reliable estimates, a well-poised sample set and an appropriate global surrogate. An adaptive model calibration scheme is designed to tackle these three issues correspondingly. First, a systematic parameter estimation approach is developed to ensure reliable estimates via heuristic subset selection consisting of estimability analysis and reliability evaluation. To capture the parameter behavior among the multiple experimental conditions and meanwhile reduce the operating cost, an adaptive experimental design is employed to guide condition testing. Experimental conditions are sequentially determined by comprehensively considering the criteria of sampling density, local nonlinearity and parameter uncertainty. Support vector regression is introduced as the global surrogate due to its capability of small-sample learning. Finally, the effectiveness of the model calibration scheme and its application performance in HTR-PM are validated by the simulation results.

Adaptive and robust experimental design for linear dynamical models using Kalman filter

Adaptive and robust experimental design for linear dynamical models using Kalman filter

AN ADAPTIVE STRATEGY FOR SEQUENTIAL DESIGNS OF MULTILEVEL COMPUTER EXPERIMENTS

Investigating uncertainties in computer simulations can be prohibitive in terms of computational costs, since the simulator needs to be run over a large number of input values. Building an emulator, i.e., a statistical surrogate model of the simulator constructed using a design of experiments made of a comparatively small number of evaluations of the forward solver, greatly alleviates the computational burden to carry out such investigations. Nevertheless, this can still be above the computational budget for many studies. Two major approaches have been used to reduce the budget needed to build the emulator: efficient design of experiments, such as sequential designs, and combining training data of different degrees of sophistication in a so-called multifidelity method, or multilevel in case these fidelities are ordered typically for increasing resolutions. We present here a novel method that combines both approaches, the multilevel adaptive sequential design of computer experiments in the framework of Gaussian process (GP) emulators. We make use of reproducing kernel Hilbert spaces as a tool for our GP approximations of the differences between two consecutive levels. This dual strategy allows us to allocate efficiently limited computational resources over simulations of different levels of fidelity and build the GP emulator. The allocation of computational resources is shown to be the solution of a simple optimization problem in a special case where we theoretically prove the validity of our approach. Our proposed method is compared to other existing models of multifidelity Gaussian process emulation. Gains in orders of magnitudes in accuracy or computing budgets are demonstrated in some numerical examples for some settings.

An adaptive information-theoretic experimental design procedure for high-to-low fidelity calibration of prostate cancer models.

The use of mathematical models to make predictions about tumor growth and response to treatment has become increasingly prevalent in the clinical setting. The level of complexity within these models ranges broadly, and the calibration of more complex models requires detailed clinical data. This raises questions about the type and quantity of data that should be collected and when, in order to maximize the information gain about the model behavior while still minimizing the total amount of data used and the time until a model can be calibrated accurately. To address these questions, we propose a Bayesian information-theoretic procedure, using an adaptive score function to determine the optimal data collection times and measurement types. The novel score function introduced in this work eliminates the need for a penalization parameter used in a previous study, while yielding model predictions that are superior to those obtained using two potential pre-determined data collection protocols for two different prostate cancer model scenarios: one in which we fit a simple ODE system to synthetic data generated from a cellular automaton model using radiotherapy as the imposed treatment, and a second scenario in which a more complex ODE system is fit to clinical patient data for patients undergoing intermittent androgen suppression therapy. We also conduct a robust analysis of the calibration results, using both error and uncertainty metrics in combination to determine when additional data acquisition may be terminated.

Adaptive experimental design produces superior and more efficient estimates of predator functional response.

Ecological dynamics are strongly influenced by the relationship between prey density and predator feeding behavior-that is, the predatory functional response. A useful understanding of this relationship requires us to distinguish between competing models of the functional response, and to robustly estimate the model parameters. Recent advances in this topic have revealed bias in model comparison, as well as in model parameter estimation in functional response studies, mainly attributed to the quality of data. Here, we propose that an adaptive experimental design framework can mitigate these challenges. We then present the first practical demonstration of the improvements it offers over standard experimental design. Our results reveal that adaptive design can efficiently identify the preferred functional response model among the competing models, and can produce much more precise posterior distributions for the estimated functional response parameters. By increasing the efficiency of experimentation, adaptive experimental design will lead to reduced logistical burden.

Deep Gaussian Processes for Calibration of Computer Models (with Discussion)

Bayesian calibration of black-box computer models offers an established framework for quantification of uncertainty of model parameters and predictions. Traditional Bayesian calibration involves the emulation of the computer model and an additive model discrepancy term using Gaussian processes; inference is then carried out using Markov chain Monte Carlo. This calibration approach is limited by the poor scalability of Gaussian processes and by the need to specify a sensible covariance function to deal with the complexity of the computer model and the discrepancy. In this work, we propose a novel calibration framework, where these challenges are addressed by means of compositions of Gaussian processes into Deep Gaussian processes and scalable variational inference techniques. Thanks to this formulation, it is possible to obtain a flexible calibration approach, which is easy to implement in development environments featuring automatic differentiation and exploiting GPU-type hardware. We show how our proposal yields a powerful alternative to the state-of-the-art by means of experimental validations on various calibration problems. We conclude the paper by showing how we can carry out adaptive experimental design, and by discussing the identifiability properties of the proposed calibration model.

Design of Molecules with Low Hole and Electron Reorganization Energy Using DFT Calculations and Bayesian Optimization.

Materials exhibiting higher mobility than conventional organic semiconducting materials, such as fullerenes and fused thiophenes, are in high demand for applications in printed electronics. To discover new molecules that might show improved charge mobility, the adaptive design of experiments (DoE) to design molecules with low reorganization energy was performed by combining density functional theory (DFT) methods and machine learning techniques. DFT-calculated values of 165 molecules were used as an initial training dataset for a Gaussian process regression (GPR) model, and five rounds of molecular designs applying the GPR model and validation via DFT calculations were executed. As a result, new molecules whose reorganization energy is smaller than the lowest value in the initial training dataset were successfully discovered.

Adaptive Design of Experiments for Safety Evaluation of Automated Vehicles

Automated Vehicles (AVs) need to be thoroughly evaluated in order to ensure their driving capabilities. However, comprehensive evaluations are intractable due to both time and monetary costs. To address this problem, we propose an Adaptive Design of Experiments (ADOE) method to evaluate the safety of AVs. Using this method, a Surrogate Model (SM) is established and updated iteratively. SM in ADOE is used to approximate the results of AV testing and help to select the next concrete scenario to be tested in each iteration. Two different ADOE approaches are proposed in this study for different testing purposes. Since the choice of the surrogate model has a profound impact on the performance of the ADOE method, 6 surrogate models were compared with two logical scenarios at different scales – a car following logical scenario and a cut-in logical scenario. Results show that Extreme Gradient Boosting (XGB) is suitable for both ADOE approaches. And both proposed ADOE approaches achieved desired performance. Scenario-oriented ADOE made full use of each concrete scenario, capturing one collision case for every 1.12 test runs in the car following logical scenario, while SM-oriented ADOE successfully depicted the boundary between safety and danger. Using 0.46% test resources compared to enumeration, the SM-oriented ADOE found 93.9% dangerous scenarios with 90.9% precision. ADOE approaches have great potential in accelerating the evaluation of AV safety.