Expensive Black-box Functions Research Articles

Bayesian Optimal Design of Experiments for Inferring the Statistical Expectation of Expensive Black-Box Functions

Abstract Bayesian optimal design of experiments (BODEs) have been successful in acquiring information about a quantity of interest (QoI) which depends on a black-box function. BODE is characterized by sequentially querying the function at specific designs selected by an infill-sampling criterion. However, most current BODE methods operate in specific contexts like optimization, or learning a universal representation of the black-box function. The objective of this paper is to design a BODE for estimating the statistical expectation of a physical response surface. This QoI is omnipresent in uncertainty propagation and design under uncertainty problems. Our hypothesis is that an optimal BODE should be maximizing the expected information gain in the QoI. We represent the information gain from a hypothetical experiment as the Kullback–Liebler (KL) divergence between the prior and the posterior probability distributions of the QoI. The prior distribution of the QoI is conditioned on the observed data, and the posterior distribution of the QoI is conditioned on the observed data and a hypothetical experiment. The main contribution of this paper is the derivation of a semi-analytic mathematical formula for the expected information gain about the statistical expectation of a physical response. The developed BODE is validated on synthetic functions with varying number of input-dimensions. We demonstrate the performance of the methodology on a steel wire manufacturing problem.

Batch Bayesian optimization using multi-scale search

Hyperparameters are crucial to many machine learning algorithms. Conventional methods like grid search can be expensive when tuning hyperparameters of a complex model, for example, deep neural networks. Recently, Bayesian optimization has become popular in the machine learning community as an efficient tool for tuning hyperparameters. Bayesian optimization is a global optimization technique that is well suited for optimizing expensive black-box functions. Traditionally Bayesian optimization operates sequentially with one recommendation per trial. However, often a batch of recommendations can be simultaneously evaluated. Current batch Bayesian methods are mostly heuristic based and none have considered heteroscedastic nature of the unknown objective functions. This paper proposes a new batch Bayesian optimization method using a multi-scale search strategy. A batch of recommendations is constructed by searching for optimal recommendations across objective functions using alternate smoothness assumptions. Theoretical analysis shows that the proposed batch improves the regret bound of the sequential method by an order of K, where K is the batch size. We show the effectiveness of our techniques by minimizing three benchmark global optimization test functions and tuning the hyperparameters of two machine learning algorithms.

Parallel Simultaneous Perturbation Optimization

Stochastic computer simulations enable users to gain new insights into complex physical systems. Optimization is a common problem in this context: users seek to find model inputs that maximize the expected value of an objective function. The objective function, however, is time-intensive to evaluate, and cannot be directly measured. Instead, the stochastic nature of the model means that individual realizations are corrupted by noise. More formally, we consider the problem of optimizing the expected value of an expensive black-box function with continuously-differentiable mean, from which observations are corrupted by Gaussian noise. We present parallel simultaneous perturbation optimization (PSPO), which extends a well-known stochastic optimization algorithm, simultaneous perturbation stochastic approximation, in several important ways. Our modifications allow the algorithm to fully take advantage of parallel computing resources, like high-performance cloud computing. The resulting PSPO algorithm takes fewer time-consuming iterations to converge, automatically chooses the step size, and can vary the error tolerance by step. Theoretical results are supported by a numerical example.

Selection of a covariance function for a Gaussian random field aimed for modeling global optimization problems

Bayesian approach is actively used to develop global optimization algorithms aimed at expensive black box functions. One of the challenges in this approach is the selection of an appropriate model for the objective function. Normally, a Gaussian random field is chosen as a theoretical model. However, the problem of estimation of parameters, using objective function values, is not thoroughly researched. In this paper, we consider the behavior of the maximum likelihood estimators of parameters of the homogeneous isotropic Gaussian random field with squared exponential covariance function. We also compare properties of exponential covariance function models.

A Trust-Region Algorithm for Heterogeneous Multiobjective Optimization

This paper presents a trust-region method for multiobjective heterogeneous optimization problems. One of the objective functions is an expensive black-box function, given, for example, by a time-consuming simulation. For this function, derivative information cannot be used, and the computation of function values involves high computational effort. The other objective functions are given analytically, and derivatives can easily be computed. The method uses the basic trust-region approach by restricting the computations in every iteration to a local area and replacing the objective functions by suitable models. The search direction is generated in the image space by using local ideal points. The algorithm generates a sequence of iterates. It is proved that any limit point is Pareto critical. Numerical results are presented and compared to two other algorithms.

Gaussian Process-Based Dimension Reduction for Goal-Oriented Sequential Design

Several methods are available for goal-oriented sequential design of expensive black-box functions. Yet, it is a difficult task when the dimension increases. A classical approach is two-stage. Firs...

GPareto: An R Package for Gaussian-Process-Based Multi-Objective Optimization and Analysis

The GPareto package for R provides multi-objective optimization algorithms for expensive black-box functions and an ensemble of dedicated uncertainty quantification methods. Popular methods such as efficient global optimization in the mono-objective case rely on Gaussian processes or kriging to build surrogate models. Driven by the prediction uncertainty given by these models, several infill criteria have also been proposed in a multi-objective setup to select new points sequentially and efficiently cope with severely limited evaluation budgets. They are implemented in the package, in addition with Pareto front estimation and uncertainty quantification visualization in the design and objective spaces. Finally, it attempts to fill the gap between expert use of the corresponding methods and user-friendliness, where many efforts have been put on providing graphical postprocessing, standard tuning and interactivity.

A reliability-based optimization method using sequential surrogate model and Monte Carlo simulation

This paper presents a sequential surrogate model method for reliability-based optimization (SSRBO), which aims to reduce the number of the expensive black-box function calls in reliability-based optimization. The proposed method consists of three key steps. First, the initial samples are selected to construct radial basis function surrogate models for the objective and constraint functions, respectively. Second, by solving a series of special optimization problems in terms of the surrogate models, local samples are identified and added in the vicinity of the current optimal point to refine the surrogate models. Third, by solving the optimization problem with the shifted constraints, the current optimal point is obtained. Then, at the current optimal point, the Monte Carlo simulation based on the surrogate models is carried out to obtain the cumulative distribution functions (CDFs) of the constraints. The CDFs and target reliabilities are used to update the offsets of the constraints for the next iteration. Therefore, the original problem is decomposed to serial cheap surrogate-based deterministic problems and Monte Carlo simulations. Several examples are adopted to verify SSRBO. The results show that the number of the expensive black-box function calls is reduced exponentially without losing of precision compared to the alternative methods, which illustrates the efficiency and accuracy of the proposed method.

Enhanced surrogate assisted framework for constrained global optimization of expensive black-box functions

An enhanced surrogate assisted framework, based on Probability of Improvement (PI) method, is proposed in this paper. We made some modifications to the original PI approach to enhance the performance of the modeling and optimization framework, leading to fewer rigorous simulations to find the optimal solution without loss of accuracy. We also extended the algorithm for handling general constraints using a fully probabilistic approach. The behavior of the proposed framework was investigated through a set of 9 Unconstrained Test Functions (UTF), 7 Constrained Optimization Problems (COP) and 3 Chemical Engineering Problems (CEP). The numerical results indicate that a lower number of rigorous model simulations were needed for optimizing UTF compared to the classic PI method and that the proposed framework was capable of achieving sustained near optimal solutions for COP and CEP. These results indicate that the proposed framework is suitable for solving computationally expensive constrained black-box optimization problems.

Filtering Bayesian optimization approach in weakly specified search space

Bayesian optimization (BO) has recently emerged as a powerful and flexible tool for hyper-parameter tuning and more generally for the efficient global optimization of expensive black-box functions. Systems implementing BO have successfully solved difficult problems in automatic design choices and machine learning hyper-parameters tunings. Many recent advances in the methodologies and theories underlying Bayesian optimization have extended the framework to new applications and provided greater insights into the behavior of these algorithms. Still, these established techniques always require a user-defined space to perform optimization. This pre-defined space specifies the ranges of hyper-parameter values. In many situations, however, it can be difficult to prescribe such spaces, as a prior knowledge is often unavailable. Setting these regions arbitrarily can lead to inefficient optimization—if a space is too large, we can miss the optimum with a limited budget, and on the other hand, if a space is too small, it may not contain the optimum point that we want to get. The unknown search space problem is intractable to solve in practice. Therefore, in this paper, we narrow down to consider specifically the setting of “weakly specified” search space for Bayesian optimization. By weakly specified space, we mean that the pre-defined space is placed at a sufficiently good region so that the optimization can expand and reach to the optimum. However, this pre-defined space need not include the global optimum. We tackle this problem by proposing the filtering expansion strategy for Bayesian optimization. Our approach starts from the initial region and gradually expands the search space. We develop an efficient algorithm for this strategy and derive its regret bound. These theoretical results are complemented by an extensive set of experiments on benchmark functions and two real-world applications which demonstrate the benefits of our proposed approach.

An Adaptive Aggregation-Based Approach for Expensively Constrained Black-Box Optimization Problems

Expensive constraints are commonly seen in real-world engineering design. However, metamodel based design optimization (MBDO) approaches often assume inexpensive constraints. In this work, the situational adaptive Kreisselmeier and Steinhauser (SAKS) method was employed in the development of a hybrid adaptive aggregation-based constraint handling strategy for expensive black-box constraint functions. The SAKS method is a novel approach that hybridizes the modeling and aggregation of expensive constraints and adds an adaptive strategy to control the level of hybridization. The SAKS strategy was integrated with a modified trust region-based mode pursuing sampling (TRMPS) algorithm to form the SAKS-trust region optimizer (SAKS-TRO) for single-objective design optimization problems with expensive black-box objective and constraint functions. SAKS-TRO was benchmarked against five popular constrained optimizers and demonstrated superior performance on average. SAKS-TRO was also applied to optimize the design of an industrial recessed impeller.

Optimal design of a parallel beam system with elastic supports to minimize flexural response to harmonic loading using a combined optimization algorithm

Mechanical systems subject to vibration are prevalent across many industries. Although potentially different in application, they sometimes share the need to minimize aspects of flexural deformation given harmonic loading and the need to consider a variety of both input and response-based constraints in the process. Practical design efforts also sometimes include the need for consideration of the optimal response of a platform-style product, including responses of multiple design variants supported by a common base structure. Harmonic problems can be especially challenging to optimize due to the likelihood that the response will be multi-modal; influenced by system natural frequencies throughout the design space. Further, analysis of these systems often involves large and complex computer models which require significant resources to execute. A harmonically loaded, platform-style parallel beam system with multiple family variants is used as an example in this work to demonstrate a proposed method for identifying an optimum in a constrained, multi-modal response environment with consideration for Expensive Black Box Functions (EBBF). The presented method proposes a combined approach where the high modality, EBBF’s domain is first surveyed for potential areas of optimal response using a method of Steepest Feasible Descent (SFD), followed by a local search in the optimal region using a more efficient direct search method. The method of SFD is a modification of the classical method of Steepest Descent, made useful for constrained models by a penalty system including both deterministic and programmatic methods. A sensitivity-based search vector method also helps to manage situations where significant difference in magnitude exists among the design variables. Evidentiary support for these key program elements is provided using standardized test functions. The effectiveness of the method is demonstrated by seeking a minimum flexural response for a parallel beam system subject to elastic support and response constraints.

Surrogate-based global optimization using an adaptive switching infill sampling criterion for expensive black-box functions

Surrogate-based global optimization algorithms use a surrogate model along with a sampling criterion. The AMP-SBGO algorithm sequentially samples points to gradually find the global optimum. The sampling criterion used to decide where to sample in each iteration dominates the algorithm and directly impacts its efficiency and robustness. This paper presents a method that uses multiple criteria for each phase of sampling, with conditions for switching from one criterion to another. Such behavior can improve the performance of the algorithm by allowing the optimization process to be less influenced by the initial sample points. Each phase, referred to as the global search phase and local search phase, utilizes different techniques. For the global search, a weighted maximin distance metric is proposed that is more efficient than ordinary maximin distance searches, and for the local search, the surrogate is optimized using a multi-start gradient-based optimizer. The algorithm was tested on 9 unconstrained mathematical test functions and 4 classes of GKLS functions along with 5 constrained test problems, which included 4 engineering design problems, and showed significant improvements compared to existing surrogate-based global optimization algorithms. The algorithm was then implemented to optimize the shape of a flange shaft in a washing machine.

Hybrid surrogate-based optimization using space reduction (HSOSR) for expensive black-box functions

In this paper, a surrogate-based global optimization algorithm HSOSR is presented, which can solve expensive black-box optimization problems with box constraints. In order to decrease difficulty of the search in large-scale multimodal problems, a space reduction method based on hybrid surrogates is proposed. Kriging and Radial Basis Function (RBF) are employed to approximate the true expensive problems, respectively. A large number of samples are generated by Latin hypercube sampling to obtain the predictive values from the two surrogates. According to the size of these predictive values from kriging and RBF, all the samples are sorted, respectively. Subsequently, two potentially better regions from kriging and RBF are identified based on the ranks of these samples and two subspaces are also created. Since kriging and RBF models always produce multiple predictive optimal solutions, a multi-start optimization algorithm is proposed to capture the supplementary samples in the two subspaces alternately. Besides, the newly added samples need to satisfy a defined distance criterion for sampling diversity. Once the algorithm gets stuck in a local valley, the estimated mean square error of kriging is maximized by the multi-start optimization strategy to explore the sparsely sampled area. Eventually, 10 low-dimensional and 5 high-dimensional benchmark cases are used to test HSOSR. In addition, 5 surrogate-based global optimization algorithms are also tested as contrast. Compared with the well-known efficient global optimization (EGO) method, HSOSR achieves an improvement of more than 50% on computational efficiency. To sum up, HSOSR has the high efficiency and strong robustness in dealing with multimodal expensive black-box optimization problems.

SOCEMO: Surrogate Optimization of Computationally Expensive Multiobjective Problems

We present the algorithm SOCEMO for optimization problems that have multiple conflicting computationally expensive black-box objective functions. The computational expense arising from the objective function evaluations considerably restricts the number of evaluations that can be done to find Pareto-optimal solutions. Frequently used multiobjective optimization methods are based on evolutionary strategies and generally require a prohibitively large number of function evaluations to find a good approximation of the Pareto front. SOCEMO, in contrast, employs surrogate models to approximate the expensive objective functions. These surrogate models are used in the iterative sampling process to decide at which points in the variable domain the next expensive evaluations should be done. Therefore, fewer expensive objective function evaluations are needed, and a good approximation of the Pareto front can be found efficiently. Previous algorithms have generally been tested on problems with few variables (up to 10) and few objective functions (up to 5). In our numerical study, we show that our algorithm performs well for benchmark problems with up to 35 dimensions and up to 10 objective functions, as well as two engineering application problems. We compared the performance of SOCEMO to a variant of NSGA-II and show that SOCEMO’s sophisticated search strategy is more efficient than NSGA-II when the number of allowable function evaluations is low. The online supplement is available at https://doi.org/10.1287/ijoc.2017.0749 .

Machine–Learning in Optimization of Expensive Black–Box Functions

Abstract Modern engineering design optimization often uses computer simulations to evaluate candidate designs. For some of these designs the simulation can fail for an unknown reason, which in turn may hamper the optimization process. To handle such scenarios more effectively, this study proposes the integration of classifiers, borrowed from the domain of machine learning, into the optimization process. Several implementations of the proposed approach are described. An extensive set of numerical experiments shows that the proposed approach improves search effectiveness.

A modified DIRECT algorithm for hidden constraints in an LNG process optimization

Optimization for process design in the chemical engineering industry has been important for energy efficiency and economic feasibility. Because many industries perform optimization with a commercial process simulator such as the Aspen HYSYS, an optimization methodology for expensive black-box functions is needed. Thus, the development of derivative free optimization algorithms has long been studied and the deterministic global search algorithm DIRECT (DIviding a hyper-RECTangle) was suggested. In this paper, a modified DIRECT algorithm with a sub-dividing step for considering hidden constraints is proposed. The effectiveness of the algorithm is exemplified by its application to a cryogenic mixed refrigerant process using a single mixed refrigerant for natural gas liquefaction and its comparison with a well-known stochastic algorithm (GA, PSO, SA), and model based search algorithm (SNOBFIT), local solver (GPS, GSS, MADS, active-set, interior-point, SQP), and other hidden constraint handling methods, including the barrier approach and the neighborhood assignment strategy. Optimal solution calculated by the proposed algorithms decreases the specific power required for natural gas liquefaction to 18.9% compared to the base case.

고비용 블랙박스 함수의 RBF기반 근사 최적화 기법

고비용 블랙박스 함수의 RBF기반 근사 최적화 기법

Extending Expected Improvement for High-Dimensional Stochastic Optimization of Expensive Black-Box Functions

Design optimization under uncertainty is notoriously difficult when the objective function is expensive to evaluate. State-of-the-art techniques, e.g., stochastic optimization or sampling average approximation, fail to learn exploitable patterns from collected data and require a lot of objective function evaluations. There is a need for techniques that alleviate the high cost of information acquisition and select sequential simulations optimally. In the field of deterministic single-objective unconstrained global optimization, the Bayesian global optimization (BGO) approach has been relatively successful in addressing the information acquisition problem. BGO builds a probabilistic surrogate of the expensive objective function and uses it to define an information acquisition function (IAF) that quantifies the merit of making new objective evaluations. In this work, we reformulate the expected improvement (EI) IAF to filter out parametric and measurement uncertainties. We bypass the curse of dimensionality, since the method does not require learning the response surface as a function of the stochastic parameters, and we employ a fully Bayesian interpretation of Gaussian processes (GPs) by constructing a particle approximation of the posterior of its hyperparameters using adaptive Markov chain Monte Carlo (MCMC) to increase the methods robustness. Also, our approach quantifies the epistemic uncertainty on the location of the optimum and the optimal value as induced by the limited number of objective evaluations used in obtaining it. We verify and validate our approach by solving two synthetic optimization problems under uncertainty and demonstrate it by solving the oil-well placement problem (OWPP) with uncertainties in the permeability field and the oil price time series.

A Kriging-based constrained global optimization algorithm for expensive black-box functions with infeasible initial points

In many engineering optimization problems, the objective and the constraints which come from complex analytical models are often black-box functions with extensive computational effort. In this case, it is necessary for optimization process to use sampling data to fit surrogate models so as to reduce the number of objective and constraint evaluations as soon as possible. In addition, it is sometimes difficult for the constrained optimization problems based on surrogate models to find a feasible point, which is the premise of further searching for a global optimal feasible solution. For this purpose, a new Kriging-based Constrained Global Optimization (KCGO) algorithm is proposed. Unlike previous Kriging-based methods, this algorithm can dispose black-box constrained optimization problem even if all initial sampling points are infeasible. There are two pivotal phases in KCGO algorithm. The main task of the first phase is to find a feasible point when there is no feasible data in the initial sample. And the aim of the second phase is to obtain a better feasible point under the circumstances of fewer expensive function evaluations. Several numerical problems and three design problems are tested to illustrate the feasibility, stability and effectiveness of the proposed method.