Black-box Optimization Problems Research Articles

High-Dimensional Bayesian Optimization via Tree-Structured Additive Models

Bayesian Optimization (BO) has shown significant success in tackling expensive low-dimensional black-box optimization problems. Many optimization problems of interest are high-dimensional, and scaling BO to such settings remains an important challenge. In this paper, we consider generalized additive models in which low-dimensional functions with overlapping subsets of variables are composed to model a high-dimensional target function. Our goal is to lower the computational resources required and facilitate faster model learning by reducing the model complexity while retaining the sample-efficiency of existing methods. Specifically, we constrain the underlying dependency graphs to tree structures in order to facilitate both the structure learning and optimization of the acquisition function. For the former, we propose a hybrid graph learning algorithm based on Gibbs sampling and mutation. In addition, we propose a novel zooming-based algorithm that permits generalized additive models to be employed more efficiently in the case of continuous domains. We demonstrate and discuss the efficacy of our approach via a range of experiments on synthetic functions and real-world datasets.

Mercer Features for Efficient Combinatorial Bayesian Optimization

Bayesian optimization (BO) is an efficient framework for solving black-box optimization problems with expensive function evaluations. This paper addresses the BO problem setting for combinatorial spaces (e.g., sequences and graphs) that occurs naturally in science and engineering applications. A prototypical example is molecular optimization guided by expensive experiments. The key challenge is to balance the complexity of statistical models and tractability of search to select combinatorial structures for evaluation. In this paper, we propose an efficient approach referred as Mercer Features for Combinatorial Bayesian Optimization (MerCBO). The key idea behind MerCBO is to provide explicit feature maps for diffusion kernels over discrete objects by exploiting the structure of their combinatorial graph representation. These Mercer features combined with Thompson sampling as the acquisition function allows us to employ efficient solvers for finding the next structure for evaluation. Experimental evaluation on diverse real-world benchmarks demonstrates that MerCBO performs similarly or better than prior methods.

Parallel Adaptive Kriging Method with Constraint Aggregation for Expensive Black-Box Optimization Problems

Design optimization problems with black-box computation-intensive objective and constraints are extremely challenging in engineering practices. To address this issue, an efficient metamodel-based optimization strategy using parallel adaptive kriging method with constraint aggregation (PAKM-CA) is proposed. In PAKM-CA, the complex expensive constraints are aggregated using the Kreisselmeier and Steinhauser (KS) function. Besides, based on the notion of Pareto nondomination in terms of objective optimality and KS function feasibility, a novel parallel comprehensive feasible expected improvement (PCFEI) function considering the correlations of sample points is developed to effectively determine the sequential infill sample points. The infill sample points with the highest PCFEI function values are selected to dynamically refine the kriging metamodels, which simultaneously improves the optimality and feasibility of optimization. Moreover, the optimization time can be further reduced via the parallel sampling framework of PCFEI. Then the convergence and efficiency merits of PAKM-CA are demonstrated via comparing with competitive state-of-the-art metamodel-based constrained optimization methods on numerical benchmarks. Finally, PAKM-CA is applied to a practical long-range slender guided rocket multidisciplinary design optimization problem to illustrate its effectiveness and practicality for solving real-world engineering problems.

Operational Risk Reverse Stress Testing: Optimal Solutions

Selecting a suitable method to solve a black-box optimization problem that uses noisy data was considered. A targeted stop condition for the function to be optimized, implemented as a stochastic algorithm, makes established Bayesian methods inadmissible. A simple modification was proposed and shown to improve optimization the efficiency considerably. The optimization effectiveness was measured in terms of the mean and standard deviation of the number of function evaluations required to achieve the target. Comparisons with alternative methods showed that the modified Bayesian method and binary search were both performant, but in different ways. In a sequence of identical runs, the former had a lower expected value for the number of runs needed to find an optimal value. The latter had a lower standard deviation for the same sequence of runs. Additionally, we suggested a way to find an approximate solution to the same problem using symbolic computation. Faster results could be obtained at the expense of some impaired accuracy and increased memory requirements.

Kriging-assisted constrained optimization of single-mixed refrigerant natural gas liquefaction process

This paper presents a surrogate-assisted framework to solve constrained black-box optimization problems, which is applied to the optimal single-mixed refrigerant natural gas liquefaction processes design. This approach uses kriging models as cheap-to-evaluate, reliable surrogates of the process-simulator-dependent black-box functions of the optimization problem. From kriging prediction and estimated error, the probability of feasible improvement acquisition function is derived and optimized to search forfeasible and improving candidates to perform local search in the penalized simulation. The kriging-assisted global search scheme is efficient to find promising candidates for local refinement, and it improves previous results that used only local optimization by 3.36 and 5.32%. The surrogate model reduces the time to evaluate the functions from 0.9933 ± 0.0815 to 3.515e-4 ± 8.353e-5 s. The kriging approximation mean squared errors assessed in a k-fold cross-validation is 7.271e-7 ± 1.297e-7 for the objective function and 0.3099 ± 0.0856 for the constraint.

Manifold-Inspired Search-Based Algorithm for Automated Test Case Generation

Automated test case generation based on path coverage (ATCG-PC) is a black-box optimization problem whose difficulty is attributed to the one-to-many relationship between path and test cases. It results in a large number of redundant function evaluations in the search process of algorithms for ATCG-PC. To minimize the redundant function evaluations for solving ATCG-PC, the equivalent mapping subspaces are defined to decompose the search space according to the paths. Inspired by the data distribution hypothesis, we assume that the target path can be covered by searching in one neighborhood of a test case (equivalent mapping subspace) instead of the whole search space. This paper presents a manifold-inspired search-based algorithm that finds the equivalent mapping subspaces with the test-case-path relationship matrix. Furthermore, the algorithm generates test cases covering all possible paths by searching in the found subspaces. The experimental results show the proposed algorithm has significantly lower function evaluation consumption than the state-of-the-art algorithms with the highest path coverage rate in two open-source toolkits and 16 open-source real-world programs. Several orders of magnitude of function evaluations can be saved by searching in the found equivalent mapping subspaces instead of exploring the whole search space.

Determining the optimal piecewise constant approximation for the nonhomogeneous Poisson process rate of Emergency Department patient arrivals

Modeling the arrival process to an Emergency Department (ED) is the first step of all studies dealing with the patient flow within the ED. Many of them focus on the increasing phenomenon of ED overcrowding, which is afflicting hospitals all over the world. Since Discrete Event Simulation models are often adopted to assess solutions for reducing the impact of this problem, proper nonstationary processes are taken into account to reproduce time–dependent arrivals. Accordingly, an accurate estimation of the unknown arrival rate is required to guarantee the reliability of results. In this work, an integer nonlinear black–box optimization problem is solved to determine the best piecewise constant approximation of the time-varying arrival rate function, by finding the optimal partition of the 24 h into a suitable number of not equally spaced intervals. The black-box constraints of the optimization problem make the feasible solutions satisfy proper statistical hypotheses; these ensure the validity of the nonhomogeneous Poisson assumption about the arrival process, commonly adopted in the literature, and prevent mixing overdispersed data for model estimation. The cost function of the optimization problem includes a fit error term for the solution accuracy and a penalty term to select an adequate degree of regularity of the optimal solution. To show the effectiveness of this methodology, real data from one of the largest Italian hospital EDs are used.

Trust-region based adaptive radial basis function algorithm for global optimization of expensive constrained black-box problems

It has been a very challenging task to develop efficient and robust techniques to solve real-world engineering optimization problems due to the unknown function properties, complex constraints and severely limited computational budget. To address this issue, TARBF algorithm (trust-region based adaptive radial basis function interpolation) for solving expensive constrained black-box optimization problems is proposed in this paper. The approach successfully decomposes the original optimization problem into a sequence of sub-problems approximated by radial basis functions in a series of trust regions. Then, the solution of each sub-problem becomes the starting point for the next iteration. According to the values of objective and constraint functions, an effective online normalization technique is further developed to adaptively improve the model accuracy in the trust region, where the surrogate is updated iteratively. Averagely, TARBF has the ability to robustly solve the 21 G-problems (CEC’2006) and 4 engineering problems within 535.69 and 234.44 function evaluations, respectively. The comparison results with other state-of-the-art metamodel-based algorithms prove that TARBF is a convergent, efficient and accurate paradigm. Moreover, the sophisticated trust region strategy developed in TARBF, which is a major contribution to the field of the efficient constrained optimization, has the capability to facilitate an effective balance of exploration and exploitation for solving constrained black-box optimization problems.

Analyzing randomness effects on the reliability of exploratory landscape analysis

The inherent difficulty of solving a continuous, static, bound-constrained and single-objective black-box optimization problem depends on the characteristics of the problem’s fitness landscape and the algorithm being used. Exploratory landscape analysis (ELA) uses numerical features generated via a sampling process of the search space to describe such characteristics. Despite their success in a number of applications, these features have limitations related with the computational costs associated with generating accurate results. Consequently, only approximations are available in practice which may be unreliable, leading to systemic errors. The overarching aim of this paper is to evaluate the reliability of five well-known ELA feature sets across multiple dimensions and sample sizes. For this purpose, we propose a comprehensive experimental methodology combining exploratory and statistical validation stages, which uses resampling techniques to minimize the sampling cost, and statistical significance tests to identify strengths and weaknesses of individual features. The data resulting from the methodology is collected and made available in the LEarning and OPtimization Archive of Research Data v1.0. The results show that instances of the same function can have feature values that are significantly different; hence, non-generalizable across instances, due to the effects produced by the boundary constraints. In addition, some landscape features under evaluation are highly volatile, and strongly susceptible to changes in sample size. Finally, the results show evidence of a curse of modality, meaning that the sample size should increase with the number of local optima.

Improved Gaussian Process Acquisition for Targeted Bayesian Optimization

A black-box optimization problem is considered, in which the function to be optimized can only be expressed in terms of a complicated stochastic algorithm that takes a long time to evaluate. The value returned is required to be sufficiently near to a target value, and uses data that has a significant noise component. Bayesian Optimization with an underlying Gaussian Process is used as an optimization solution, and its effectiveness is measured in terms of the number of function evaluations required to attain the target. To improve results, a simple modification of the Gaussian Process ‘Lower Confidence Bound’ (LCB) acquisition function is proposed. The expression used for the confidence bound is squared in order to better comply with the target requirement. With this modification, much improved results compared to random selection methods and to other commonly used acquisition functions are obtained.

A Bayesian Optimization Approach to Decentralized Event-Triggered Control

In this paper, we investigate a model-free design of decentralized event-triggered mechanism for networked control systems (NCSs). The approach aims at simultaneously tuning the optimal parameters for the controller and the event-triggered condition, such that a prescribed cost function can be minimized. To achieve this goal, we employ the Bayesian optimization (BO), which is known to be an automatic tuning framework for finding the optimal solution to the black-box optimization problem. Thanks to its efficient search strategy for the global optimum, the BO allows us to design the event-triggered mechanism with relatively a small number of experimental evaluations. This is particularly suited for NCSs where network resources such as the limited life-time of battery powered devices are limited. Some simulation examples illustrate the effectiveness of the approach.

Using the iCC framework for solving unconstrained LSGO problems

Evolutionary computation is a branch of computational intelligence that is inspired by nature. This algorithm family uses fundamental rules of natural selection to find a better solution when solving optimization problems. The productivity of these methods decreases when the number of parameters to be optimized is extra-large. This area of research is known as large-scale global optimization. Cooperative coevolution is a structure for evolutionary algorithms which tries to solve the curse of dimensionality problem. The cooperative coevolution efficiency of application largely depends on the two settings: the size of each subcomponent(subproblem) and the grouping of variables. The actual work proposes improved cooperative coevolution denoted as iCC. The iCC approach dynamically resizes the groups. iCC starts with a predefined set of subproblems and reduces them gradually during the optimization process. A novel metaheuristic has been developed which is called iCC-SHADE for black-box optimization problems with a large number of variables. The proposed method has been tested on fifteen optimization tasks from the LSGO CEC’2013 competition benchmark. The experimental results have demonstrated that iCC-SHADE has statistically better performance than CC-SHADE with a static number of subproblems. Also, the effectiveness of iCC-SHADE has been tested in comparison with other modern metaheuristics. The Wilcoxon rank-sum test was used to compare the effectiveness of investigated metaheuristics.

An Estimation of Distribution Algorithm With Filtering and Learning

Estimation of distribution algorithm (EDA) is an efficient population-based stochastic search technique. Since it was proposed, many attempts have been made to improve its performance in the context of nonlinear continuous optimization. However, the success of EDA depends on the accuracy of modeling, the effectiveness of sampling, and the ability of exploration. An effective EDA often needs to take some measures to adjust the model and to guide sampling. In this article, we propose a novel EDA which applies the idea of Kalman filtering to revise the modeling data and a learning strategy to improve sampling. The filtering scheme modifies the modeling data set using an estimation error matrix based on historic solution data. During the sampling process, the learning strategy determines the region to sample next based on the sampling outcomes so far, instead of completely random sampling. The proposed EDA also employs a multivariate probabilistic model based on copula function and can quickly reach the promising area in which the optimal solution is likely to be located. A collection of general benchmark functions are used to test the performance of the proposed algorithm. Computational experiments show that the EDA is effective. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —In many process industries, there exist black-box operation optimization problems and large-scale nonlinear optimization problems with variable coupling. For these problems, it is difficult to establish mechanism models between input and output. However, real-time data can be measured from the system through sensors. We can utilize this process information to optimize the system so as to attain the desired objective. In this article, we propose a novel estimation of distribution algorithm (EDA) which applies a filtering scheme to revise the modeling data and a learning strategy to improve sampling, which can solve the problems with the characteristics of nonlinearity, variable coupling, and large scale. Computational experiments show that the EDA is effective. In the future, the proposed algorithm can be applied to some practical optimization problems such as operation optimization in blast furnace, which is considered as a continuous production process with variable coupling. The algorithm has the potential to help optimizing the process control parameters.

A Data-Driven Automatic Tuning Method for MPC under Uncertainty using Constrained Bayesian Optimization

Abstract The closed-loop performance of model predictive controllers (MPCs) is highly dependent on the choice of prediction models, controller formulation, and tuning parameters. However, prediction models are typically optimized for prediction accuracy, instead of performance, and MPC tuning is typically done manually to satisfy (probabilistic) constraints. In this work, we demonstrate a general approach for automating the tuning of MPC under uncertainty. In particular, we formulate the automated tuning problem as a constrained black-box optimization problem that can be tackled with derivative-free optimization. We rely on a constrained variant of Bayesian optimization to solve the MPC tuning problem that can directly handle noisy and expensive-to-evaluate functions. The benefits of the proposed automated tuning approach are demonstrated on a benchmark continuously stirred tank reactor case study.

Dynamic Neighborhood-Based Particle Swarm Optimization for Multimodal Problems

Various black-box optimization problems in real world can be classified as multimodal optimization problems. Neighborhood information plays an important role in improving the performance of an evolutionary algorithm when dealing with such problems. In view of this, we propose a particle swarm optimization algorithm based on dynamic neighborhood to solve the multimodal optimization problem. In this paper, a dynamic ε-neighborhood selection mechanism is first defined to balance the exploration and exploitation of the algorithm. Then, based on the information provided by the neighborhoods, four different particle position updating strategies are designed to further support the algorithm’s exploration and exploitation of the search space. Finally, the proposed algorithm is compared with 7 state-of-the-art multimodal algorithms on 8 benchmark instances. The experimental results reveal that the proposed algorithm is superior to the compared ones and is an effective method to tackle multimodal optimization problems.

Landscape-Aware Performance Prediction for Evolutionary Multiobjective Optimization

We expose and contrast the impact of landscape characteristics on the performance of search heuristics for black-box multiobjective combinatorial optimization problems. A sound and concise summary of features characterizing the structure of an arbitrary problem instance is identified and related to the expected performance of global and local dominance-based multiobjective optimization algorithms. We provide a critical review of existing features tailored to multiobjective combinatorial optimization problems, and we propose additional ones that do not require any global knowledge from the landscape, making them suitable for large-size problem instances. Their intercorrelation and their association with algorithm performance are also analyzed. This allows us to assess the individual and the joint effect of problem features on algorithm performance, and to highlight the main difficulties encountered by such search heuristics. By providing effective tools for multiobjective landscape analysis, we highlight that multiple features are required to capture problem difficulty, and we provide further insights into the importance of ruggedness and multimodality to characterize multiobjective combinatorial landscapes.

Data-Driven Scenario Optimization for Automated Controller Tuning With Probabilistic Performance Guarantees

Systematic design and verification of advanced control strategies for complex systems under uncertainty largely remains an open problem. Despite the promise of black-box optimization methods for automated controller tuning, they generally lack formal guarantees on the solution quality, which is especially important in the control of safety-critical systems. This letter focuses on obtaining closed-loop performance guarantees for automated controller tuning, which can be formulated as a black-box optimization problem under uncertainty. We use recent advances in non-convex scenario theory to provide a distribution-free bound on the probability of the closed-loop performance measures. To mitigate the computational complexity of the data-driven scenario optimization method, we restrict ourselves to a discrete set of candidate tuning parameters. We propose to generate these candidates using constrained Bayesian optimization run multiple times from different random seed points. We apply the proposed method for tuning an economic nonlinear model predictive controller for a semibatch reactor modeled by seven highly nonlinear differential equations.

Surrogate-assisted teaching-learning-based optimization for high-dimensional and computationally expensive problems

In this work, a surrogate-assisted teaching-learning-based optimization algorithm is presented for high-dimensional and computationally expensive black-box optimization problems. In the presented method, a two-phase searching framework is proposed to realize the exploitation of surrogates and the metaheuristic exploration. Specifically, radial basis functions are used to build the dynamically updated surrogate models. Moreover, a surrogate-assisted knowledge mining strategy is proposed to sufficiently collect valuable information in each cycle. In this strategy, multiple groups of promising points are selected from the expensive sample set to construct the subspaces and local surrogate models, from which the most potential points are captured for the subsequent updates and optimization. At the same time, a population composed of the present best expensive samples carries out the teaching/learning-based search, accelerating local convergence and promoting global exploration. In the surrogate-assisted teaching phase, both of individual and mean behaviors are considered to make learners go close to the teacher efficiently. In the surrogate-assisted learning phase, a more extensive search range is defined to improve sampling diversity. The cooperation of the surrogate-assisted knowledge mining and the modified teaching-learning-based exploration makes the new method have excellent performance on 21 benchmark cases with 30–100 design variables. Furthermore, this method is used for the shape optimization of a blended-wing-body underwater glider, and gets impressive results.

Bi-space Interactive Cooperative Coevolutionary algorithm for large scale black-box optimization

Large scale black-box optimization problems arise in many fields of science and engineering, and many of existing algorithms for these problems still suffer from the “curse of dimensionality”. This paper proposes a generalized framework of Bi-space Interactive Cooperative Coevolutionary Algorithm (BICCA) with evolutions in two spaces. In the pattern space, the interacting patterns of variables are continuously excavated for the evolution of the groups for cooperative coevolution. In the search space, cooperative coevolution and global search are carried out adaptively to get better fitness. By adopting evolutions and interactions within two spaces, patterns evolve to provide better groupings while individuals evolve to reach better fitness. The problem decomposition is conducted along the optimization process, and no extra fitness evaluations are needed for problem decomposition. Experiments on widely-used benchmarks show that BICCA obtains competitive performance on high-dimensional optimization problems with different levels of dimensionality up to 10000.

Hyperparameter optimization for recommender systems through Bayesian optimization

Recommender systems represent one of the most successful applications of machine learning in B2C online services, to help the users in their choices in many web services. Recommender system aims to predict the user preferences from a huge amount of data, basically the past behaviour of the user, using an efficient prediction algorithm. One of the most used is the matrix-factorization algorithm. Like many machine learning algorithms, its effectiveness goes through the tuning of its hyper-parameters, and the associated optimization problem also called hyper-parameter optimization. This represents a noisy time-consuming black-box optimization problem. The related objective function maps any possible hyper-parameter configuration to a numeric score quantifying the algorithm performance. In this work, we show how Bayesian optimization can help the tuning of three hyper-parameters: the number of latent factors, the regularization parameter, and the learning rate. Numerical results are obtained on a benchmark problem and show that Bayesian optimization obtains a better result than the default setting of the hyper-parameters and the random search.