Black-box Objective Function Research Articles

Solving higher dimensional expensive black box global optimization problems using sparse directional search on surrogates

We present a new algorithm Directional Optimization Search with Surrogate (DOSS), for optimizing problems with box constraints and a computationally expensive black-box objective function. DOSS is a radial basis function (RBF) based method that mainly focuses on higher dimensional and computationally expensive objective functions that can be multimodal. DOSS introduces three new techniques not previously used in earlier RBF algorithms, including using a combination of the coordinates knowledge level, fewer initial sampling points, and the surrogate’s gradient information. Numerical results on a test set including 14 test problems with 36, 48, and 60 dimensions show that DOSS outperforms two recently published algorithms RBFOpt and TuRBO, and earlier RBF algorithms such as DYCORS. TuRBO is a Gaussian process based optimization algorithm, which outperformed earlier state-of-the-art methods. DOSS algorithm also has a good performance on real-world optimization problems, including robot pushing and rover trajectory planning problems. Almost sure convergence for the DOSS algorithm is also proven in this paper. An implementation of DOSS is available at https://doi.org/10.5281/zenodo.13731558.

Managing low–acuity patients in an Emergency Department through simulation–based multiobjective optimization using a neural network metamodel

This paper deals with Emergency Department (ED) fast-tracks for low-acuity patients, a strategy often adopted to reduce ED overcrowding. We focus on optimizing resource allocation in minor injuries units, which are the ED units that can treat low-acuity patients, with the aim of minimizing patient waiting times and ED operating costs. We formulate this problem as a general multiobjective simulation-based optimization problem where some of the objectives are expensive black-box functions that can only be evaluated through a time-consuming simulation. To efficiently solve this problem, we propose a metamodeling approach that uses an artificial neural network to replace a black-box objective function with a suitable model. This approach allows us to obtain a set of Pareto optimal points for the multiobjective problem we consider, from which decision-makers can select the most appropriate solutions for different situations. We present the results of computational experiments conducted on a real case study involving the ED of a large hospital in Italy. The results show the reliability and effectiveness of our proposed approach, compared to the standard approach based on derivative-free optimization.

A physics informed bayesian optimization approach for material design: application to NiTi shape memory alloys

The design of materials and identification of optimal processing parameters constitute a complex and challenging task, necessitating efficient utilization of available data. Bayesian Optimization (BO) has gained popularity in materials design due to its ability to work with minimal data. However, many BO-based frameworks predominantly rely on statistical information, in the form of input-output data, and assume black-box objective functions. In practice, designers often possess knowledge of the underlying physical laws governing a material system, rendering the objective function not entirely black-box, as some information is partially observable. In this study, we propose a physics-informed BO approach that integrates physics-infused kernels to effectively leverage both statistical and physical information in the decision-making process. We demonstrate that this method significantly improves decision-making efficiency and enables more data-efficient BO. The applicability of this approach is showcased through the design of NiTi shape memory alloys, where the optimal processing parameters are identified to maximize the transformation temperature.

Exploration of new chemical materials using black-box optimization with the D-wave quantum annealer

In materials informatics, searching for chemical materials with desired properties is challenging due to the vastness of the chemical space. Moreover, the high cost of evaluating properties necessitates a search with a few clues. In practice, there is also a demand for proposing compositions that are easily synthesizable. In the real world, such as in the exploration of chemical materials, it is common to encounter problems targeting black-box objective functions where formalizing the objective function in explicit form is challenging, and the evaluation cost is high. In recent research, a Bayesian optimization method has been proposed to formulate the quadratic unconstrained binary optimization (QUBO) problem as a surrogate model for black-box objective functions with discrete variables. Regarding this method, studies have been conducted using the D-Wave quantum annealer to optimize the acquisition function, which is based on the surrogate model and determines the next exploration point for the black-box objective function. In this paper, we address optimizing a black-box objective function containing discrete variables in the context of actual chemical material exploration. In this optimization problem, we demonstrate results obtaining parameters of the acquisition function by sampling from a probability distribution with variance can explore the solution space more extensively than in the case of no variance. As a result, we found combinations of substituents in compositions with the desired properties, which could only be discovered when we set an appropriate variance.

Hybrid quantum ResNet for car classification and its hyperparameter optimization

Image recognition is one of the primary applications of machine learning algorithms. Nevertheless, machine learning models used in modern image recognition systems consist of millions of parameters that usually require significant computational time to be adjusted. Moreover, adjustment of model hyperparameters leads to additional overhead. Because of this, new developments in machine learning models and hyperparameter optimization techniques are required. This paper presents a quantum-inspired hyperparameter optimization technique and a hybrid quantum-classical machine learning model for supervised learning. We benchmark our hyperparameter optimization method over standard black-box objective functions and observe performance improvements in the form of reduced expected run times and fitness in response to the growth in the size of the search space. We test our approaches in a car image classification task and demonstrate a full-scale implementation of the hybrid quantum ResNet model with the tensor train hyperparameter optimization. Our tests show a qualitative and quantitative advantage over the corresponding standard classical tabular grid search approach used with a deep neural network ResNet34. A classification accuracy of 0.97 was obtained by the hybrid model after 18 iterations, whereas the classical model achieved an accuracy of 0.92 after 75 iterations.

Dependence in constrained Bayesian optimization

Constrained Bayesian optimization optimizes a black-box objective function subject to black-box constraints. For simplicity, most existing works assume that multiple constraints are independent. To ask, when and how does dependence between constraints help?, we remove this assumption and implement probability of feasibility with dependence (Dep-PoF) by applying multiple output Gaussian processes (MOGPs) as surrogate models and using expectation propagation to approximate the probabilities. We compare Dep-PoF and the independent version PoF. We propose two new acquisition functions incorporating Dep-PoF and test them on synthetic and practical benchmarks. Our results are largely negative: incorporating dependence between the constraints does not help much. Empirically, incorporating dependence between constraints may be useful if: (i) the solution is on the boundary of the feasible region(s) or (ii) the feasible set is very small. When these conditions are satisfied, the predictive covariance matrix from the MOGP may be poorly approximated by a diagonal matrix and the off-diagonal matrix elements may become important. Dep-PoF may apply to settings where (i) the constraints and their dependence are totally unknown and (ii) experiments are so expensive that any slightly better Bayesian optimization procedure is preferred. But, in most cases, Dep-PoF is indistinguishable from PoF.

Benchmarking surrogate-based optimisation algorithms on expensive black-box functions

Surrogate algorithms such as Bayesian optimisation are especially designed for black-box optimisation problems with expensive objectives, such as hyperparameter tuning or simulation-based optimisation. In the literature, these algorithms are usually evaluated with synthetic benchmarks which are well established but have no expensive objective, and only on one or two real-life applications which vary wildly between papers. There is a clear lack of standardisation when it comes to benchmarking surrogate algorithms on real-life, expensive, black-box objective functions. This makes it very difficult to draw conclusions on the effect of algorithmic contributions and to give substantial advice on which method to use when. A new benchmark library, EXPObench, provides first steps towards such a standardisation. The library is used to provide an extensive comparison of six different surrogate algorithms on four expensive optimisation problems from different real-life applications. This has led to new insights regarding the relative importance of exploration, the evaluation time of the objective, and the used model. We also provide rules of thumb for which surrogate algorithm to use in which situation. A further contribution is that we make the algorithms and benchmark problem instances publicly available, contributing to more uniform analysis of surrogate algorithms. Most importantly, we include the results of the six algorithms on all evaluated problem instances. This unique new dataset lowers the bar for researching new methods as the number of expensive evaluations required for comparison and for the creation of new surrogate models is significantly reduced.

Bayesian Optimization of Queuing-Based Multichannel URLLC Scheduling

This paper studies the allocation of shared resources between ultra-reliable low-latency communication (URLLC) and enhanced mobile broadband (eMBB) in the emerging 5G and beyond cellular networks. In this paper, we design a unique queuing mechanism for the joint eMBB/URLLC system. The aim is to flexibly schedule URLLC traffic to enhance the total eMBB throughput and the reliability of URLLC packets (i.e., the probability of not dropping URLLC packets in each mini-slot) while maintaining a satisfactory transmission latency as per the 3GPP requirements. Precisely, by deriving the steady-state probabilities of URLLC queue backlog analytically, we formulate a stochastic optimization problem to maximize the total normalized eMBB throughput and the URLLC utility. Due to the stochastic nature of the objective function, it is expensive to evaluate it for any set of inputs, and thus the Bayesian optimization is applied to obtain the optimal results of such a black-box objective function. Numerical results demonstrate that the proposed queuing mechanism never violates the latency requirement of the URLLC services but improves the reliability. It also enhances the total normalized eMBB throughput as compared to the method without queuing.

Deep Kernel Learning-Based Bayesian Optimization with Adaptive Kernel Functions

Bayesian optimization (BO) is a widely used data-driven method for the global optimization of black-box objective functions with noise. A key component of BO is surrogate model that is a probabilistic model used for approximating black-box objective functions. However, classical BO methods often utilize Gaussian processes with stationary covariance functions as surrogate models, which is difficult to accurately model complex systems with rapidly oscillating target functions, and therefore deteriorates the performance of BO. To this end, a Bayesian optimization approach based on deep kernel learning (DKL) is investigated for the optimization of complex objective functions with large oscillations, in which a Gaussian process based on deep kernel learning is utilized to capture the characteristics of data with non-stationary and hierarchical covariance functions via deep neural networks. The optimization performance of the DKL-based BO approach with different network structures is quantified and compared with a state-of-the-art BO method based on Gaussian process under the scenarios of different noise levels.

A Surrogate-Based Optimization Method with Dynamic Adaptation for High-Dimensional Mixed-Integer Problems

• SODA-AD and its two variants are proposed to address HMIO-B under limited computational resources. • SODA-AD and its two variants can well balance global and local search. • SODA-AD and its two variants outperform other counterpart algorithms. This study develops a Surrogate-based Optimization algorithm with Dynamic Adaptation of perturbation search of All Dimensions (SODA-AD) to address high-dimensional mixed-integer optimization problems with a black-box objective function (HMIO-B). SODA-AD improves dynamic coordinate search (DCS) in two ways, i.e., with a new way of sampling candidate points and a new adaptive scaling method, which help to balance global and local search and address high-dimensional problems. Additionally, two variants of SODA-AD are described. One is SODA-AD with a modified infill strategy (SODA-ADM), which uses the prediction scoring criterion to replace the weighted scoring criterion when the budgeted computation resources are going to run out. The second method first employs SODA-ADM and then carries out a sequential dimensioned perturbation search for each iteration periodically to continue the local search, and this is named SODA-ADM-DP. In numerical experiments, we compare SODA-AD and its two variants with other well-known counterparts using one complex real-world engineering problem and eight 100-dimensional (100-D) benchmark problems. It is concluded that SODA-AD and its two variants outperform the other counterparts on most of the test problems and are promising for solving high-dimensional mixed-integer optimization problems with black-box objective functions or nonlinear and nonconvex objective functions.

Cylinder drag minimization through wall actuation: A Bayesian Optimization approach

Bayesian Optimization (BO) has recently gained popularity as an efficient derivative-free method for the global optimization of expensive noisy black-box objective functions. These characteristics render BO a promising tool to tackle optimization problems involving numerical simulations of complex unsteady flows at moderate-to-high Reynolds numbers. In this work, we assess the efficiency of Bayesian Optimization by considering two canonical flow problems: the drag reduction in the two-dimensional and three-dimensional flow around circular cylinders at, respectively, Re=500 and Re=3900, through tangential-velocity actuation at the cylinder wall. The root-mean-square of the drag coefficient with and without penalty terms is considered as the objective function. Several variants of Bayesian Optimization are assessed and compared against competing optimization algorithms such as Particle Swarm Optimization, CMA-ES, Nelder–Mead and the Explorative Gradient Method. Results show that in this case, the serial and the parallel BO techniques outperform other algorithms.

Framework for embedding black-box simulation into mathematical programming via kriging surrogate model applied to natural gas liquefaction process optimization

This paper presents a framework to solve the constrained black-box simulation optimization problem that arises from the optimal energy-efficient design of single-mixed refrigerant natural gas liquefaction process using reliable process simulator. Kriging surrogate model is used to introduce simple, computationally inexpensive, and effective algebraic formulations with reliable derivatives to the black-box objective and constraints functions. The algebraic surrogate optimization problem is embedded into a nonlinear programming (NLP) model in General Algebraic Modeling System (GAMS). The NLP problem is solved using efficient multi-start gradient-based optimization with CONOPT local solver to determine a candidate of decision variables for which the true functions are calculated in the rigorous simulation. The single-mixed refrigerant process is analyzed considering one-to-three-stage expansion and phase separation to assess potential energy savings. The present approach results show that more expansion stages can provide energy savings from 12.02 to 14.70 % comparing two-stage and three-stage expansion system with single-stage. This optimization framework is more effective and consistent than Particle Swarm Optimization and Genetic Algorithm given the same budget of simulation evaluations for the considered simulation optimization problems, resulting in 13.57 to 53.26 % of energy savings.

ADVANCED ALGORITHM OF EVOLUTION STRATEGIES OF COVARIATION MATRIX ADAPTATION

The paper considers the extension of the CMA-ES algorithm using mixtures of distributions for finding optimal hyperparameters of neural networks. Hyperparameter optimization, formulated as the optimization of the black box objective function, which is a necessary condition for automation and high performance of machine learning approaches. CMA-ES is an efficient optimization algorithm without derivatives, one of the alternatives in the combination of hyperparameter optimization methods. The developed algorithm is based on the assumption of a multi-peak density distribution of the parameters of complex systems. Compared to other optimization methods, CMA-ES is computationally inexpensive and supports parallel computations. Research results show that CMA-ES can be competitive, especially in the concurrent assessment mode. However, a much broader and more detailed comparison is still needed, which will include more test tasks and various modifications, such as adding constraints. Based on the Monte Carlo method, it was shown that the new algorithm will improve the search for optimal hyperparameters by an average of 12%.

Framework for Embedding Process Simulator in GAMS via Kriging Surrogate Model Applied to C3MR Natural Gas Liquefaction Optimization

Rigorous black-box simulations are useful to describe complex systems. However, it cannot be directly integrated into mathematical programming models in some algebraic modeling environments because of the lack of symbolic formulation. In the present paper, a framework is proposed to embed the Aspen HYSYS process simulator in GAMS using kriging surrogate models to replace the simulator-dependent, black-box objective, and constraints functions. The approach is applied to the energy-efficient C3MR natural gas liquefaction process simulation optimization using multi-start nonlinear programming and the local solver CONOPT in GAMS. Results were compared with two other meta-heuristic approaches, Particle Swarm Optimization (PSO) and Genetic Algorithm (GA), and with the literature. In a small simulation evaluation budget of 20 times the number of decision variables, the proposed optimization approach resulted in 0.2538 kW of compression work per kg of natural gas and surpassed those of the PSO and GA and the previous literature from 2.45 to 15.3 %.

A Multi-Objective Bayesian Optimization Approach Using the Weighted Tchebycheff Method

Abstract Bayesian optimization (BO) is a low-cost global optimization tool for expensive black-box objective functions, where we learn from prior evaluated designs, update a posterior surrogate Gaussian process model, and select new designs for future evaluation using an acquisition function. This research focuses upon developing a BO model with multiple black-box objective functions. In the standard multi-objective (MO) optimization problem, the weighted Tchebycheff method is efficiently used to find both convex and non-convex Pareto frontiers. This approach requires knowledge of utopia values before we start optimization. However, in the BO framework, since the functions are expensive to evaluate, it is very expensive to obtain the utopia values as a prior knowledge. Therefore, in this paper, we develop a MO-BO framework where we calibrate with multiple linear regression (MLR) models to estimate the utopia value for each objective as a function of design input variables; the models are updated iteratively with sampled training data from the proposed MO-BO. These iteratively estimated mean utopia values are used to formulate the weighted Tchebycheff MO acquisition function. The proposed approach is implemented in two numerical test examples and one engineering design problem of optimizing thin tube geometries under constant loading of temperature and pressure, with minimizing the risk of creep-fatigue failure and design cost, along with risk-based and manufacturing constraints. Finally, the model accuracy with frequentist, Bayesian and without MLR-based calibration are compared to true Pareto solutions.

Benchmark Test of Black-box Optimization Using D-Wave Quantum Annealer

In solving optimization problems, objective functions generally need to be minimized or maximized. However, objective functions cannot always be formulated explicitly in a mathematical form for complicated problem settings. Although several regression techniques infer the approximate forms of objective functions, they are at times expensive to evaluate. Optimal points of "black-box" objective functions are computed in such scenarios, while effectively using a small number of clues. Recently, an efficient method by use of inference by sparse prior for a black-box objective function with binary variables has been proposed. In this method, a surrogate model was proposed in the form of a quadratic unconstrained binary optimization (QUBO) problem, and was iteratively solved to obtain the optimal solution of the black-box objective function. In the present study, we employ the D-Wave 2000Q quantum annealer, which can solve QUBO by driving the binary variables by quantum fluctuations. The D-Wave 2000Q quantum annealer does not necessarily output the ground state at the end of the protocol due to freezing effect during the process. We investigate effects from the output of the D-Wave quantum annealer in performing black-box optimization. We demonstrate a benchmark test by employing the sparse Sherrington-Kirkpatrick (SK) model as the black-box objective function, by introducing a parameter controlling the sparseness of the interaction coefficients. Comparing the results of the D-Wave quantum annealer to those of the simulated annealing (SA) and semidefinite programming (SDP), our results by the D-Wave quantum annealer and SA exhibit superiority in black-box optimization with SDP. On the other hand, we did not find any advantage of the D-Wave quantum annealer over the simulated annealing. As far as in our case, any effects by quantum fluctuation are not found.

ITSO: a novel inverse transform sampling-based optimization algorithm for stochastic search

Optimization algorithms appear in the core calculations of numerous Artificial Intelligence (AI) and Machine Learning methods and Engineering and Business applications. Following recent works on AI’s theoretical deficiencies, a rigour context for the optimization problem of a black-box objective function is developed. The algorithm stems directly from the theory of probability, instead of presumed inspiration. Thus the convergence properties of the proposed methodology are inherently stable. In particular, the proposed optimizer utilizes an algorithmic implementation of the n-dimensional inverse transform sampling as a search strategy. No control parameters are required to be tuned, and the trade-off among exploration and exploitation is, by definition, satisfied. A theoretical proof is provided, concluding that when falling into the proposed framework, either directly or incidentally, any optimization algorithm converges. The numerical experiments verify the theoretical results on the efficacy of the algorithm apropos reaching the sought optimum.

A principled approach to design using high fidelity fluid-structure interaction simulations

A high fidelity fluid-structure interaction simulation may require many days to run, on hundreds of cores. This poses a serious burden, both in terms of time and economic considerations, when repetitions of such simulations may be required (e.g. for the purpose of design optimization). In this paper we present strategies based on (constrained) Bayesian optimization (BO) to alleviate this burden. BO is a numerical optimization technique based on Gaussian processes (GP) that is able to efficiently (with minimal calls to the expensive FSI models) converge towards some globally optimal design, as gauged using a black box objective function. In this study we present a principled design evolution that moves from FSI model verification, through a series of Bridge Simulations (bringing the verification case incrementally closer to the application), in order that we may identify material properties for an underwater, unmanned, autonomous vehicle (UUAV) sail plane. We are able to achieve fast convergence towards an optimal design, using a small number of FSI simulations (a dozen at most), even when selecting over several design parameters, and while respecting optimization constraints.

Automated Design of Deep Neural Networks

In recent years, research in applying optimization approaches in the automatic design of deep neural networks has become increasingly popular. Although various approaches have been proposed, there is a lack of a comprehensive survey and taxonomy on this hot research topic. In this article, we propose a unified way to describe the various optimization algorithms that focus on common and important search components of optimization algorithms: representation, objective function, constraints, initial solution(s), and variation operators. In addition to large-scale search space, the problem is characterized by its variable mixed design space, it is very expensive, and it has multiple blackbox objective functions. Hence, this unified methodology has been extended to advanced optimization approaches, such as surrogate-based, multi-objective, and parallel optimization.

Calibrating Path Choices and Train Capacities for Urban Rail Transit Simulation Models Using Smart Card and Train Movement Data

Transit network simulation models are often used for performance and retrospective analysis of urban rail systems, taking advantage of the availability of extensive automated fare collection (AFC) and automated vehicle location (AVL) data. Important inputs to such models, in addition to origin-destination flows, include passenger path choices and train capacity. Train capacity, which has often been overlooked in the literature, is an important input that exhibits a lot of variabilities. The paper proposes a simulation-based optimization (SBO) framework to simultaneously calibrate path choices and train capacity for urban rail systems using AFC and AVL data. The calibration is formulated as an optimization problem with a black-box objective function. Seven algorithms from four branches of SBO solving methods are evaluated. The algorithms are evaluated using an experimental design that includes five scenarios, representing different degrees of path choice randomness and crowding sensitivity. Data from the Hong Kong Mass Transit Railway (MTR) system is used as a case study. The data is used to generate synthetic observations used as “ground truth.” The results show that the response surface methods (particularly constrained optimization using response surfaces) have consistently good performance under all scenarios. The proposed approach drives large-scale simulation applications for monitoring and planning.