Optimization Of Expensive Black-box Functions Research Articles

Constrained multiobjective optimization of expensive black-box functions using a heuristic branch-and-bound approach

Constrained multiobjective optimization of expensive black-box functions using a heuristic branch-and-bound approach

Ability of Black-Box Optimisation to Efficiently Perform Simulation Studies in Power Engineering

Abstract In this study, the potential of the so-called black-box optimisation (BBO) to increase the efficiency of simulation studies in power engineering is evaluated. Three algorithms (“Multilevel Coordinate Search” (MCS) and “Stable Noisy Optimization by Branch and Fit” (SNOBFIT) by Huyer and Neumaier and “blackbox: A Procedure for Parallel Optimization of Expensive Black-box Functions” (blackbox) by Knysh and Korkolis) are implemented in MATLAB and compared for solving two use cases: the analysis of the maximum rotational speed of a gas turbine after a load rejection and the identification of transfer function parameters by measurements. The first use case has a high computational cost, whereas the second use case is computationally cheap. For each run of the algorithms, the accuracy of the found solution and the number of simulations or function evaluations needed to determine the optimum and the overall runtime are used to identify the potential of the algorithms in comparison to currently used methods. All methods provide solutions for potential optima that are at least 99.8% accurate compared to the reference methods. The number of evaluations of the objective functions differs significantly but cannot be directly compared as only the SNOBFIT algorithm does stop when the found solution does not improve further, whereas the other algorithms use a predefined number of function evaluations. Therefore, SNOBFIT has the shortest runtime for both examples. For computationally expensive simulations, it is shown that parallelisation of the function evaluations (SNOBFIT and blackbox) and quantisation of the input variables (SNOBFIT) are essential for the algorithmic performance. For the gas turbine overspeed analysis, only SNOBFIT can compete with the reference procedure concerning the runtime. Further studies will have to investigate whether the quantisation of input variables can be applied to other algorithms and whether the BBO algorithms can outperform the reference methods for problems with a higher dimensionality.

Bayesian optimization for a multiple-component system with target values

Bayesian optimization (BO) that employs the Gaussian process (GP) as a surrogate model has recently gained much attention in optimization of expensive black-box functions. In BO, the number of experiments necessary to optimize a function can be considerably reduced by sequentially selecting next design points that are optimal with respect to some sampling criterion. However, little research has been done to address the optimization of a multiple-component system where each component has a certain target value to meet. In this paper, we aim to find an optimal design parameter in the sense that the response function is close to the target value for every component. To this end, the squared errors from the targets are aggregated to produce an objective function. Instead of modeling this objective using GP as in the standard BO formulation, we place the GP prior over the response function. As a result, the distribution over the objective function follows that of the weighted sum of non-central chi-squared random variables (WSNC) due to the inter-dependency between responses. When components of the system are changed, the standard BO suffers inefficiency; however, our formulation enables us to retain a learned model, resulting in better efficiency. We compare the rates of convergence of different BO methods and other black-box optimization baselines using several test functions. The performance of our model is comparable to the standard BO when there is no change in the system, but the superiority of our method becomes clear when changes in the components occur.

Enhanced surrogate assisted framework for constrained global optimization of expensive black-box functions

An enhanced surrogate assisted framework, based on Probability of Improvement (PI) method, is proposed in this paper. We made some modifications to the original PI approach to enhance the performance of the modeling and optimization framework, leading to fewer rigorous simulations to find the optimal solution without loss of accuracy. We also extended the algorithm for handling general constraints using a fully probabilistic approach. The behavior of the proposed framework was investigated through a set of 9 Unconstrained Test Functions (UTF), 7 Constrained Optimization Problems (COP) and 3 Chemical Engineering Problems (CEP). The numerical results indicate that a lower number of rigorous model simulations were needed for optimizing UTF compared to the classic PI method and that the proposed framework was capable of achieving sustained near optimal solutions for COP and CEP. These results indicate that the proposed framework is suitable for solving computationally expensive constrained black-box optimization problems.

Filtering Bayesian optimization approach in weakly specified search space

Bayesian optimization (BO) has recently emerged as a powerful and flexible tool for hyper-parameter tuning and more generally for the efficient global optimization of expensive black-box functions. Systems implementing BO have successfully solved difficult problems in automatic design choices and machine learning hyper-parameters tunings. Many recent advances in the methodologies and theories underlying Bayesian optimization have extended the framework to new applications and provided greater insights into the behavior of these algorithms. Still, these established techniques always require a user-defined space to perform optimization. This pre-defined space specifies the ranges of hyper-parameter values. In many situations, however, it can be difficult to prescribe such spaces, as a prior knowledge is often unavailable. Setting these regions arbitrarily can lead to inefficient optimization—if a space is too large, we can miss the optimum with a limited budget, and on the other hand, if a space is too small, it may not contain the optimum point that we want to get. The unknown search space problem is intractable to solve in practice. Therefore, in this paper, we narrow down to consider specifically the setting of “weakly specified” search space for Bayesian optimization. By weakly specified space, we mean that the pre-defined space is placed at a sufficiently good region so that the optimization can expand and reach to the optimum. However, this pre-defined space need not include the global optimum. We tackle this problem by proposing the filtering expansion strategy for Bayesian optimization. Our approach starts from the initial region and gradually expands the search space. We develop an efficient algorithm for this strategy and derive its regret bound. These theoretical results are complemented by an extensive set of experiments on benchmark functions and two real-world applications which demonstrate the benefits of our proposed approach.

Extending Expected Improvement for High-Dimensional Stochastic Optimization of Expensive Black-Box Functions

Design optimization under uncertainty is notoriously difficult when the objective function is expensive to evaluate. State-of-the-art techniques, e.g., stochastic optimization or sampling average approximation, fail to learn exploitable patterns from collected data and require a lot of objective function evaluations. There is a need for techniques that alleviate the high cost of information acquisition and select sequential simulations optimally. In the field of deterministic single-objective unconstrained global optimization, the Bayesian global optimization (BGO) approach has been relatively successful in addressing the information acquisition problem. BGO builds a probabilistic surrogate of the expensive objective function and uses it to define an information acquisition function (IAF) that quantifies the merit of making new objective evaluations. In this work, we reformulate the expected improvement (EI) IAF to filter out parametric and measurement uncertainties. We bypass the curse of dimensionality, since the method does not require learning the response surface as a function of the stochastic parameters, and we employ a fully Bayesian interpretation of Gaussian processes (GPs) by constructing a particle approximation of the posterior of its hyperparameters using adaptive Markov chain Monte Carlo (MCMC) to increase the methods robustness. Also, our approach quantifies the epistemic uncertainty on the location of the optimum and the optimal value as induced by the limited number of objective evaluations used in obtaining it. We verify and validate our approach by solving two synthetic optimization problems under uncertainty and demonstrate it by solving the oil-well placement problem (OWPP) with uncertainties in the permeability field and the oil price time series.

Value-Based Global Optimization

In this paper, a value-based global optimization (VGO) algorithm is introduced. The algorithm uses kriging-like surrogate models and a sequential sampling strategy based on value of information (VoI) to optimize an objective characterized by multiple analysis models with different accuracies. VGO builds on two main contributions. The first contribution is a novel surrogate modeling method that accommodates data from any number of different analysis models with varying accuracy and cost. Rather than interpolating, it fits a model to the data, giving more weight to more accurate data. The second contribution is the use of VoI as a new metric for guiding the sequential sampling process for global optimization. Based on information about the cost and accuracy of each available model, predictions from the current surrogate model are used to determine where to sample next and with what level of accuracy. The cost of further analysis is explicitly taken into account during the optimization process, and no further analysis occurs if the expected value of the new information is negative. In this paper, we present the details of the VGO algorithm and, using a suite of randomly generated test cases, compare its performance with the performance of the efficient global optimization (EGO) algorithm (Jones, D. R., Matthias, S., and Welch, W. J., 1998, “Efficient Global Optimization of Expensive Black-Box Functions,” J. Global Optim., 13(4), pp. 455–492). Results indicate that the VGO algorithm performs better than EGO in terms of overall expected utility—on average, the same quality solution is achieved at a lower cost, or a better solution is achieved at the same cost.

An adaptive multi-modal optimization algorithm for simulation-based design of power-electronic circuits

This article introduces a non-gradient optimization algorithm for the simulation-based design of nonlinear power-electronic circuits where multiple local minima may exist. The algorithm steers a simultaneous search for multiple optima by generating a multi-resolution mesh in the search space. It adaptively increases its resolution only in regions where the presence of local optima is likely, thus offering significant savings in the number of (simulation-based) objective-function evaluations. The algorithm's relatively low computational intensity, its independence of local gradient information and its capability to discover multiple local minima simultaneously make it appealing for the optimization of expensive black-box functions. The article shows an implementation of the algorithm in MATLAB® and links it to the PSCAD®/EMTDC transient simulation program. Examples of the application of the proposed algorithm for both analytical and simulation-based objective functions are included.

Efficient Global Optimization of Expensive Black-Box Functions

In many engineering optimization problems, the number of function evaluations is severely limited by time or cost. These problems pose a special challenge to the field of global optimization, since existing methods often require more function evaluations than can be comfortably afforded. One way to address this challenge is to fit response surfaces to data collected by evaluating the objective and constraint functions at a few points. These surfaces can then be used for visualization, tradeoff analysis, and optimization. In this paper, we introduce the reader to a response surface methodology that is especially good at modeling the nonlinear, multimodal functions that often occur in engineering. We then show how these approximating functions can be used to construct an efficient global optimization algorithm with a credible stopping rule. The key to using response surfaces for global optimization lies in balancing the need to exploit the approximating surface (by sampling where it is minimized) with the need to improve the approximation (by sampling where prediction error may be high). Striking this balance requires solving certain auxiliary problems which have previously been considered intractable, but we show how these computational obstacles can be overcome.