Batch Sequential Design Research Articles

Batch-sequential design and heteroskedastic surrogate modeling for delta smelt conservation

Delta smelt is an endangered fish species in the San Francisco estuary that have shown an overall population decline over the past 30 years. Researchers have developed a stochastic, agent-based simulator to virtualize the system with the goal of understanding the relative contribution of natural and anthropogenic factors that might play a role in their decline. However, the input configuration space is high dimensional, running the simulator is time-consuming, and its noisy outputs change nonlinearly in both mean and variance. Getting enough runs to effectively learn input–output dynamics requires both a nimble modeling strategy and parallel evaluation. Recent advances in heteroskedastic Gaussian process (HetGP) surrogate modeling helps, but little is known about how to appropriately plan experiments for highly distributed simulation. We propose a batch sequential design scheme, generalizing one-at-a-time variance-based active learning for HetGP, as a means of keeping multicore cluster nodes fully engaged with runs. Our acquisition strategy is carefully engineered to favor selection of replicates which boost statistical and computational efficiency when training surrogates to isolate signal from noise. Design and modeling are illustrated on a range of toy examples before embarking on a large-scale smelt simulation campaign and downstream high-fidelity input sensitivity analysis.

Batch sequential designs for accelerated life tests and an application to polymer composite fatigue test

Accelerated life tests (ALTs) are widely used in industry and many optimal ALT test plans are developed in literature. To achieve an efficient optimal test plan, one needs to make sure that the planning values used in the test planning are not far away from the true parameters, which is challenging because lifetime data are not collected yet in the planning stage to estimate the true parameters. To overcome the weakness, sequential designs are developed and the parameter estimates are updated during the sequential testing process. Due to the sequential nature, one drawback of the sequential strategy is that it usually requires a long total testing time (TTT). Motivated by industrial needs, we propose batch sequential design (BSD) in ALT setting, in which multiple units are tested simultaneously while sequential parameter updates are also made. The proposed BSD for ALT can not only reduce the TTT but also preserves the advantage of sequential designs. After proposing two types of BSD strategies, we use a comprehensive simulation study to demonstrate the advantages of the proposed BSD in ALT setting. We also develop a method to find the optimal batch size which is essential for the deployment of the BSD in practice.

Parallel Gaussian Process Surrogate Bayesian Inference with Noisy Likelihood Evaluations

We consider Bayesian inference when only a limited number of noisy log-likelihood evaluations can be obtained. This occurs for example when complex simulator-based statistical models are fitted to data, and synthetic likelihood (SL) method is used to form the noisy log-likelihood estimates using computationally costly forward simulations. We frame the inference task as a sequential Bayesian experimental design problem, where the log-likelihood function is modelled with a hierarchical Gaussian process (GP) surrogate model, which is used to efficiently select additional log-likelihood evaluation locations. Motivated by recent progress in the related problem of batch Bayesian optimisation, we develop various batch-sequential design strategies which allow to run some of the potentially costly simulations in parallel. We analyse the properties of the resulting method theoretically and empirically. Experiments with several toy problems and simulation models suggest that our method is robust, highly parallelisable, and sample-efficient.

Batch sequential NOAB designs by way of simultaneous construction and augmentation

Space-filling designs help experimenters to represent simulation outputs efficiently when entire input spaces cannot be exhaustively explored. Batch sequential designs allow for intermediate analyses to occur as later batches of experimental design points are being tested, given the ability to change later design points based on the outputs observed, and stop the experiment when the current observations are deemed sufficient to reduce experimental cost. Nearly orthogonal-and-balanced (NOAB) designs have good space-filling properties and can accommodate design spaces with continuous, discrete, and categorical factors. In this paper, mixed-integer linear programming (MILP) formulations used to find NOAB resolutions III, IV, and V designs are extended to construct batch sequential NOAB designs, where design stages can use different NOAB approaches. A case study is presented where a simultaneous construction approach results in overall more desirable designs than when using design augmentation, yet requires a predefined number of points for each design stage.

Efficient multiobjective optimization employing Gaussian processes, spectral sampling and a genetic algorithm

Many engineering problems require the optimization of expensive, black-box functions involving multiple conflicting criteria, such that commonly used methods like multiobjective genetic algorithms are inadequate. To tackle this problem several algorithms have been developed using surrogates. However, these often have disadvantages such as the requirement of a priori knowledge of the output functions or exponentially scaling computational cost with respect to the number of objectives. In this paper a new algorithm is proposed, TSEMO, which uses Gaussian processes as surrogates. The Gaussian processes are sampled using spectral sampling techniques to make use of Thompson sampling in conjunction with the hypervolume quality indicator and NSGA-II to choose a new evaluation point at each iteration. The reference point required for the hypervolume calculation is estimated within TSEMO. Further, a simple extension was proposed to carry out batch-sequential design. TSEMO was compared to ParEGO, an expected hypervolume implementation, and NSGA-II on nine test problems with a budget of 150 function evaluations. Overall, TSEMO shows promising performance, while giving a simple algorithm without the requirement of a priori knowledge, reduced hypervolume calculations to approach linear scaling with respect to the number of objectives, the capacity to handle noise and lastly the ability for batch-sequential usage.

Sliced Full Factorial-Based Latin Hypercube Designs as a Framework for a Batch Sequential Design Algorithm

When fitting complex models, such as finite element or discrete event simulations, the experiment design should exhibit desirable properties of both projectivity and orthogonality. To reduce experimental effort, sequential design strategies allow experimenters to collect data only until some measure of prediction precision is reached. In this article, we present a batch sequential experiment design method that uses sliced full factorial-based Latin hypercube designs (sFFLHDs), which are an extension to the concept of sliced orthogonal array-based Latin hypercube designs (OALHDs). At all stages of the sequential design, good univariate stratification is achieved. The structure of the FFLHDs also tends to produce uniformity in higher dimensions, especially at certain stages of the design. We show that our batch sequential design approach has good sampling and fitting qualities through both empirical studies and theoretical arguments. Supplementary materials are available online.

Batch Sequential Minimum Energy Design with Design-Region Adaptation

Experiments related to nanofabrication often face challenges of resource-limited experimental budgets, highly demanding tolerance requirements, and complicated response surfaces. Therefore, wisely selecting design points is crucial in order to minimize the expense of resources while at the same time ensuring that enough information is gained to accurately address the experimental goals. In this paper, an efficient batch-sequential design methodology is proposed for optimizing high-cost, low-resource experiments with complicated response surfaces. Through the sequential learning of the unknown response surface, the proposed method sequentially narrows down the design space to more important subregions and selects a batch of design points in the reduced design region. The proposed method balances the space filling of the design region and the search for the optimal operating condition. The performance of the proposed method is demonstrated on a nanowire synthesis system as well as on an optimization test function.

Bayesian improved model migration methodology for fast process modeling by incorporating prior information

We consider a Bayesian inference approach to enhance model migration, building on concepts laid out in an earlier paper (Lu and Gao, 2008a). Previous studies have been limited to a least-squares solution and have failed to take prior knowledge into consideration, possibly tending to cause overfitting and inaccurate estimations. We present a framework for Bayesian migration that can naturally incorporate and use prior information. The approach involves imposing normal-inverse-gamma priors over the migration parameter and exploring the resulting posterior distributions using a Markov chain Monte Carlo method. In addition, we provide a batch sequential design framework for iterative implementation of model migration, which thus avoids an exhaustive treatment of a predetermined number of design points. The effectiveness of these proposed methods is demonstrated using two examples: a numerical study and an injection molding process.

Batch sequential design of optimal experiments for improved predictive maturity in physics-based modeling

The focus of nuclear fuel design and maintenance has been shifting from a primarily empirical endeavor to that of highly advanced simulations characterized by experiment-based calibration, validati...

Batch sequential design to achieve predictive maturity with calibrated computer models

Sequential experiment design strategies have been proposed for efficiently augmenting initial designs to solve many problems of interest to computer experimenters, including optimization, contour and threshold estimation, and global prediction. We focus on batch sequential design strategies for achieving maturity in global prediction of discrepancy inferred from computer model calibration. Predictive maturity focuses on adding field experiments to efficiently improve discrepancy inference. Several design criteria are extended to allow batch augmentation, including integrated and maximum mean square error, maximum entropy, and two expected improvement criteria. In addition, batch versions of maximin distance and weighted distance criteria are developed. Two batch optimization algorithms are considered: modified Fedorov exchange and a binning methodology motivated by optimizing augmented fractional factorial skeleton designs.

Batch sequential designs for computer experiments

Computer models simulating a physical process are used in many areas of science. Due to the complex nature of these codes it is often necessary to approximate the code, which is typically done using a Gaussian process. In many situations the number of code runs available to build the Gaussian process approximation is limited. When the initial design is small or the underlying response surface is complicated this can lead to poor approximations of the code output. In order to improve the fit of the model, sequential design strategies must be employed. In this paper we introduce two simple distance based metrics that can be used to augment an initial design in a batch sequential manner. In addition we propose a sequential updating strategy to an orthogonal array based Latin hypercube sample. We show via various real and simulated examples that the distance metrics and the extension of the orthogonal array based Latin hypercubes work well in practice.

Hybrid methods for calculating optimal few-stage sequential strategies: Data monitoring for a clinical trial

Optimal batch-sequential designs are difficult to compute, even when sufficient statistics and relatively uncomplicated loss functions simplify the calculations required. While backward induction applies, its difficulty grows exponentially in the number of stages, while a recently developed forward algorithm grows only linearly, but involves a maximization over a rather flat surface. This paper explores a hybrid algorithm, partially backward induction, partially forward, that has some of the advantages of each.

Fully-sequential procedures in nonlinear design problems

In nonlinear design problems the major difficulty to overcome is the dependence of the optimal design points on the true vector θ of the parameters. Batch-sequential designs take advantage of the inflow of new information about θ. In this paper fully sequential procedures are discussed for any optimality criterion φ, say. Simulation studies encourange the use of a fully-sequential procedure instead of one-stage method which produce locally optimal designs.