Black-box Objective Research Articles

Cylinder drag minimization through wall actuation: A Bayesian Optimization approach

Bayesian Optimization (BO) has recently gained popularity as an efficient derivative-free method for the global optimization of expensive noisy black-box objective functions. These characteristics render BO a promising tool to tackle optimization problems involving numerical simulations of complex unsteady flows at moderate-to-high Reynolds numbers. In this work, we assess the efficiency of Bayesian Optimization by considering two canonical flow problems: the drag reduction in the two-dimensional and three-dimensional flow around circular cylinders at, respectively, Re=500 and Re=3900, through tangential-velocity actuation at the cylinder wall. The root-mean-square of the drag coefficient with and without penalty terms is considered as the objective function. Several variants of Bayesian Optimization are assessed and compared against competing optimization algorithms such as Particle Swarm Optimization, CMA-ES, Nelder–Mead and the Explorative Gradient Method. Results show that in this case, the serial and the parallel BO techniques outperform other algorithms.

Framework for embedding black-box simulation into mathematical programming via kriging surrogate model applied to natural gas liquefaction process optimization

This paper presents a framework to solve the constrained black-box simulation optimization problem that arises from the optimal energy-efficient design of single-mixed refrigerant natural gas liquefaction process using reliable process simulator. Kriging surrogate model is used to introduce simple, computationally inexpensive, and effective algebraic formulations with reliable derivatives to the black-box objective and constraints functions. The algebraic surrogate optimization problem is embedded into a nonlinear programming (NLP) model in General Algebraic Modeling System (GAMS). The NLP problem is solved using efficient multi-start gradient-based optimization with CONOPT local solver to determine a candidate of decision variables for which the true functions are calculated in the rigorous simulation. The single-mixed refrigerant process is analyzed considering one-to-three-stage expansion and phase separation to assess potential energy savings. The present approach results show that more expansion stages can provide energy savings from 12.02 to 14.70 % comparing two-stage and three-stage expansion system with single-stage. This optimization framework is more effective and consistent than Particle Swarm Optimization and Genetic Algorithm given the same budget of simulation evaluations for the considered simulation optimization problems, resulting in 13.57 to 53.26 % of energy savings.

ADVANCED ALGORITHM OF EVOLUTION STRATEGIES OF COVARIATION MATRIX ADAPTATION

The paper considers the extension of the CMA-ES algorithm using mixtures of distributions for finding optimal hyperparameters of neural networks. Hyperparameter optimization, formulated as the optimization of the black box objective function, which is a necessary condition for automation and high performance of machine learning approaches. CMA-ES is an efficient optimization algorithm without derivatives, one of the alternatives in the combination of hyperparameter optimization methods. The developed algorithm is based on the assumption of a multi-peak density distribution of the parameters of complex systems. Compared to other optimization methods, CMA-ES is computationally inexpensive and supports parallel computations. Research results show that CMA-ES can be competitive, especially in the concurrent assessment mode. However, a much broader and more detailed comparison is still needed, which will include more test tasks and various modifications, such as adding constraints. Based on the Monte Carlo method, it was shown that the new algorithm will improve the search for optimal hyperparameters by an average of 12%.

Framework for Embedding Process Simulator in GAMS via Kriging Surrogate Model Applied to C3MR Natural Gas Liquefaction Optimization

Rigorous black-box simulations are useful to describe complex systems. However, it cannot be directly integrated into mathematical programming models in some algebraic modeling environments because of the lack of symbolic formulation. In the present paper, a framework is proposed to embed the Aspen HYSYS process simulator in GAMS using kriging surrogate models to replace the simulator-dependent, black-box objective, and constraints functions. The approach is applied to the energy-efficient C3MR natural gas liquefaction process simulation optimization using multi-start nonlinear programming and the local solver CONOPT in GAMS. Results were compared with two other meta-heuristic approaches, Particle Swarm Optimization (PSO) and Genetic Algorithm (GA), and with the literature. In a small simulation evaluation budget of 20 times the number of decision variables, the proposed optimization approach resulted in 0.2538 kW of compression work per kg of natural gas and surpassed those of the PSO and GA and the previous literature from 2.45 to 15.3 %.

A Multi-Objective Bayesian Optimization Approach Using the Weighted Tchebycheff Method

Abstract Bayesian optimization (BO) is a low-cost global optimization tool for expensive black-box objective functions, where we learn from prior evaluated designs, update a posterior surrogate Gaussian process model, and select new designs for future evaluation using an acquisition function. This research focuses upon developing a BO model with multiple black-box objective functions. In the standard multi-objective (MO) optimization problem, the weighted Tchebycheff method is efficiently used to find both convex and non-convex Pareto frontiers. This approach requires knowledge of utopia values before we start optimization. However, in the BO framework, since the functions are expensive to evaluate, it is very expensive to obtain the utopia values as a prior knowledge. Therefore, in this paper, we develop a MO-BO framework where we calibrate with multiple linear regression (MLR) models to estimate the utopia value for each objective as a function of design input variables; the models are updated iteratively with sampled training data from the proposed MO-BO. These iteratively estimated mean utopia values are used to formulate the weighted Tchebycheff MO acquisition function. The proposed approach is implemented in two numerical test examples and one engineering design problem of optimizing thin tube geometries under constant loading of temperature and pressure, with minimizing the risk of creep-fatigue failure and design cost, along with risk-based and manufacturing constraints. Finally, the model accuracy with frequentist, Bayesian and without MLR-based calibration are compared to true Pareto solutions.

Benchmark Test of Black-box Optimization Using D-Wave Quantum Annealer

In solving optimization problems, objective functions generally need to be minimized or maximized. However, objective functions cannot always be formulated explicitly in a mathematical form for complicated problem settings. Although several regression techniques infer the approximate forms of objective functions, they are at times expensive to evaluate. Optimal points of "black-box" objective functions are computed in such scenarios, while effectively using a small number of clues. Recently, an efficient method by use of inference by sparse prior for a black-box objective function with binary variables has been proposed. In this method, a surrogate model was proposed in the form of a quadratic unconstrained binary optimization (QUBO) problem, and was iteratively solved to obtain the optimal solution of the black-box objective function. In the present study, we employ the D-Wave 2000Q quantum annealer, which can solve QUBO by driving the binary variables by quantum fluctuations. The D-Wave 2000Q quantum annealer does not necessarily output the ground state at the end of the protocol due to freezing effect during the process. We investigate effects from the output of the D-Wave quantum annealer in performing black-box optimization. We demonstrate a benchmark test by employing the sparse Sherrington-Kirkpatrick (SK) model as the black-box objective function, by introducing a parameter controlling the sparseness of the interaction coefficients. Comparing the results of the D-Wave quantum annealer to those of the simulated annealing (SA) and semidefinite programming (SDP), our results by the D-Wave quantum annealer and SA exhibit superiority in black-box optimization with SDP. On the other hand, we did not find any advantage of the D-Wave quantum annealer over the simulated annealing. As far as in our case, any effects by quantum fluctuation are not found.

ITSO: a novel inverse transform sampling-based optimization algorithm for stochastic search

Optimization algorithms appear in the core calculations of numerous Artificial Intelligence (AI) and Machine Learning methods and Engineering and Business applications. Following recent works on AI’s theoretical deficiencies, a rigour context for the optimization problem of a black-box objective function is developed. The algorithm stems directly from the theory of probability, instead of presumed inspiration. Thus the convergence properties of the proposed methodology are inherently stable. In particular, the proposed optimizer utilizes an algorithmic implementation of the n-dimensional inverse transform sampling as a search strategy. No control parameters are required to be tuned, and the trade-off among exploration and exploitation is, by definition, satisfied. A theoretical proof is provided, concluding that when falling into the proposed framework, either directly or incidentally, any optimization algorithm converges. The numerical experiments verify the theoretical results on the efficacy of the algorithm apropos reaching the sought optimum.

Introspective False Negative Prediction for Black-Box Object Detectors in Autonomous Driving.

Object detection plays a critical role in autonomous driving, but current state-of-the-art object detectors will inevitably fail in many driving scenes, which is unacceptable for safety-critical automated vehicles. Given the complexity of the real traffic scenarios, it is impractical to guarantee zero detection failure; thus, online failure prediction is of crucial importance to mitigate the risk of traffic accidents. Of all the failure cases, False Negative (FN) objects are most likely to cause catastrophic consequences, but little attention has been paid to the online FN prediction. In this paper, we propose a general introspection framework that can make online prediction of FN objects for black-box object detectors. In contrast to existing methods which rely on empirical assumptions or handcrafted features, we facilitate the FN feature extraction by an introspective FN predictor we designed in this framework. For this purpose, we extend the original concept of introspection to object-wise FN predictions, and propose a multi-branch cooperation mechanism to address the distinct foreground-background imbalance problem of FN objects. The effectiveness of the proposed framework is verified through extensive experiments and analysis, and the results show that our method successfully predicts the FN objects with 81.95% precision for 88.10% recall on the challenging KITTI Benchmark, and effectively improves object detection performance by taking FN predictions into consideration.

A principled approach to design using high fidelity fluid-structure interaction simulations

A high fidelity fluid-structure interaction simulation may require many days to run, on hundreds of cores. This poses a serious burden, both in terms of time and economic considerations, when repetitions of such simulations may be required (e.g. for the purpose of design optimization). In this paper we present strategies based on (constrained) Bayesian optimization (BO) to alleviate this burden. BO is a numerical optimization technique based on Gaussian processes (GP) that is able to efficiently (with minimal calls to the expensive FSI models) converge towards some globally optimal design, as gauged using a black box objective function. In this study we present a principled design evolution that moves from FSI model verification, through a series of Bridge Simulations (bringing the verification case incrementally closer to the application), in order that we may identify material properties for an underwater, unmanned, autonomous vehicle (UUAV) sail plane. We are able to achieve fast convergence towards an optimal design, using a small number of FSI simulations (a dozen at most), even when selecting over several design parameters, and while respecting optimization constraints.

Automated Design of Deep Neural Networks

In recent years, research in applying optimization approaches in the automatic design of deep neural networks has become increasingly popular. Although various approaches have been proposed, there is a lack of a comprehensive survey and taxonomy on this hot research topic. In this article, we propose a unified way to describe the various optimization algorithms that focus on common and important search components of optimization algorithms: representation, objective function, constraints, initial solution(s), and variation operators. In addition to large-scale search space, the problem is characterized by its variable mixed design space, it is very expensive, and it has multiple blackbox objective functions. Hence, this unified methodology has been extended to advanced optimization approaches, such as surrogate-based, multi-objective, and parallel optimization.

Calibrating Path Choices and Train Capacities for Urban Rail Transit Simulation Models Using Smart Card and Train Movement Data

Transit network simulation models are often used for performance and retrospective analysis of urban rail systems, taking advantage of the availability of extensive automated fare collection (AFC) and automated vehicle location (AVL) data. Important inputs to such models, in addition to origin-destination flows, include passenger path choices and train capacity. Train capacity, which has often been overlooked in the literature, is an important input that exhibits a lot of variabilities. The paper proposes a simulation-based optimization (SBO) framework to simultaneously calibrate path choices and train capacity for urban rail systems using AFC and AVL data. The calibration is formulated as an optimization problem with a black-box objective function. Seven algorithms from four branches of SBO solving methods are evaluated. The algorithms are evaluated using an experimental design that includes five scenarios, representing different degrees of path choice randomness and crowding sensitivity. Data from the Hong Kong Mass Transit Railway (MTR) system is used as a case study. The data is used to generate synthetic observations used as “ground truth.” The results show that the response surface methods (particularly constrained optimization using response surfaces) have consistently good performance under all scenarios. The proposed approach drives large-scale simulation applications for monitoring and planning.

Multifidelity Genetic Transfer: An Efficient Framework for Production Optimization

Summary Production optimization led by computing intelligence can greatly improve oilfield economic effectiveness. However, it is confronted with huge computational challenge because of the expensive black-box objective function and the high-dimensional design variables. Many low-fidelity methods based on simplified physical models or data-driven models have been proposed to reduce evaluation costs. These methods can approximate the global fitness landscape to a certain extent, but it is difficult to ensure accuracy and correlation in local areas. Multifidelity methods have been proposed to balance the advantages of the two, but most of the current methods rely on complex computational models. Through a simple but efficient shortcut, our work aims to establish a novel production-optimization framework using genetic transfer learning to accelerate convergence and improve the quality of optimal solution using results from different fidelities. Net present value (NPV) is a widely used standard to comprehensively evaluate the economic value of a strategy in production optimization. On the basis of NPV, we first established a multifidelity optimization model that can synthesize the reference information from high-fidelity tasks and the approximate results from low-fidelity tasks. Then, we introduce the concept of relative fidelity as an indicator for quantifying the dynamic reliability of low-fidelity methods, and further propose a two-mode multifidelity genetic transfer learning framework that balances computing resources for tasks with different fidelity levels. The multitasking mode takes the elite solution as the transfer medium and forms a closed-loop feedback system through the information exchange between low- and high-fidelity tasks in parallel. Sequential transfer mode, a one-way algorithm, transfers the elite solutions archived in the previous mode as the population to high-fidelity domain for further optimization. This framework is suitable for population-based optimization algorithms with variable search direction and step size. The core work of this paper is to realize the framework by means of differential evolution (DE), for which we propose the multifidelity transfer differential evolution (MTDE). Corresponding to multitasking and sequential transfer in the framework, MTDE includes two modes, transfer based on base vector (b-transfer) and transfer based on population (p-transfer). The b-transfer mode incorporates the unique advantages of DE into fidelity switching, whereas the p-transfer mode adaptively conducts population for further high-fidelity local search. Finally, the production-optimization performance of MTDE is validated with the egg model and two real field cases, in which the black-oil and streamline models are used to obtain high- and low-fidelity results, respectively. We also compared the convergence curves and optimization results with the single-fidelity method and the greedy multifidelity method. The results show that the proposed algorithm has a faster convergence rate and a higher-quality well-control strategy. The adaptive capacity of p-transfer is also demonstrated in three distinct cases. At the end of the paper, we discuss the generalization potential of the proposed framework.

GOPS: efficient RBF surrogate global optimization algorithm with high dimensions and many parallel processors including application to multimodal water quality PDE model calibration

This paper describes a new parallel global surrogate-based algorithm Global Optimization in Parallel with Surrogate (GOPS) for the minimization of continuous black-box objective functions that might have multiple local minima, are expensive to compute, and have no derivative information available. The task of picking P new evaluation points for P processors in each iteration is addressed by sampling around multiple center points at which the objective function has been previously evaluated. The GOPS algorithm improves on earlier algorithms by (a) new center points are selected based on bivariate non-dominated sorting of previously evaluated points with additional constraints to ensure the objective value is below a target percentile and (b) as iterations increase, the number of centers decreases, and the number of evaluation points per center increases. These strategies and the hyperparameters controlling them significantly improve GOPS’s parallel performance on high dimensional problems in comparison to other global optimization algorithms, especially with a larger number of processors. GOPS is tested with up to 128 processors in parallel on 14 synthetic black-box optimization benchmarking test problems (in 10, 21, and 40 dimensions) and one 21-dimensional parameter estimation problem for an expensive real-world nonlinear lake water quality model with partial differential equations that takes 22 min for each objective function evaluation. GOPS numerically significantly outperforms (especially on high dimensional problems and with larger numbers of processors) the earlier algorithms SOP and PSD-MADS-VNS (and these two algorithms have outperformed other algorithms in prior publications).

A discrete cosine transform-based query efficient attack on black-box object detectors

Deep learning models are being widely used in almost every field of computing and information processing. The advantages offered by these models are unparalleled, however, similar to any other computing discipline, they are also vulnerable to security threats. A compromised deep neural network can significantly impact its robustness and accuracy. In this work, we present a novel targeted attack method against state-of-the-art object detection models YOLO v3 and AWS Rekognition in a black-box environment. We present an improved attack method using Discrete Cosine Transform based on boundary attack plus plus mechanism, and apply it on attacking object detectors offline and online. By querying the victim detection models along with transforming the images from the spatial domain into the frequency domain, we ensure that any specified object in an image can be successfully recognized as any other desired class by YOLO v3 and AWS Rekognition. The results prove that our method has significant boosting effects on boundary attacks in offline and online object detection systems.

Human-in-the-loop differential subspace search in high-dimensional latent space

Generative models based on deep neural networks often have a high-dimensional latent space, ranging sometimes to a few hundred dimensions or even higher, which typically makes them hard for a user to explore directly. We propose differential subspace search to allow efficient iterative user exploration in such a space, without relying on domain- or data-specific assumptions. We develop a general framework to extract low-dimensional subspaces based on a local differential analysis of the generative model, such that a small change in such a subspace would provide enough change in the resulting data. We do so by applying singular value decomposition to the Jacobian of the generative model and forming a subspace with the desired dimensionality spanned by a given number of singular vectors stochastically selected on the basis of their singular values, to maintain ergodicity. We use our framework to present 1D subspaces to the user via a 1D slider interface. Starting from an initial location, the user finds a new candidate in the presented 1D subspace, which is in turn updated at the new candidate location. This process is repeated until no further improvement can be made. Numerical simulations show that our method can better optimize synthetic black-box objective functions than the alternatives that we tested. Furthermore, we conducted a user study using complex generative models and the results show that our method enables more efficient exploration of high-dimensional latent spaces than the alternatives.

Towards cross-task universal perturbation against black-box object detectors in autonomous driving

Deep neural network is the main research branch in artificial intelligence and suitable for many decision-making fields. Autonomous driving and unmanned vehicle often depend on deep neural networks for accurate and reliable detection, classification, and ranging of surrounding objects in real on-road environments, either locally or by swarm intelligence among distributed nodes via 5G channel. But, it has been demonstrated that deep neural networks are vulnerable to well-designed adversarial examples that are imperceptible to human eyes in computer vision tasks. It is valuable to study the vulnerability for enhancing the robustness of neural networks. However, existing adversarial examples against object detection models are image-dependent, so in this paper, we implement adversarial attacks against object detection models using universal perturbations. We find the cross-task, cross-model, and cross-dataset transferability of universal perturbations, we train universal perturbations generator firstly and then add the universal perturbations to the target images in two ways: resizing and pile-up, in order to solve the problem that universal perturbations cannot be directly applied to attack object detection models. We use the transferability of universal perturbations to attack black-box object detection models. In this way, the time cost of generating adversarial examples is reduced. A series of experiments are conducted on PASCAL VOC and MS COCO datasets demonstrating the feasibility of cross-task attacks and proving the effectiveness of our attack on two representative object detectors: regression-based models like YOLOv3 and proposal-based models like Faster R-CNN.

Filter-based stochastic algorithm for global optimization

We propose the general Filter-based Stochastic Algorithm (FbSA) for the global optimization of nonconvex and nonsmooth constrained problems. Under certain conditions on the probability distributions that generate the sample points, almost sure convergence is proved. In order to optimize problems with computationally expensive black-box objective functions, we develop the FbSA-RBF algorithm based on the general FbSA and assisted by Radial Basis Function (RBF) surrogate models to approximate the objective function. At each iteration, the resulting algorithm constructs/updates a surrogate model of the objective function and generates trial points using a dynamic coordinate search strategy similar to the one used in the Dynamically Dimensioned Search method. To identify a promising best trial point, a non-dominance concept based on the values of the surrogate model and the constraint violation at the trial points is used. Theoretical results concerning the sufficient conditions for the almost surely convergence of the algorithm are presented. Preliminary numerical experiments show that the FbSA-RBF is competitive when compared with other known methods in the literature.

Efficient Closed-loop Maximization of Carbon Nanotube Growth Rate using Bayesian Optimization

A major technological challenge in materials research is the large and complex parameter space, which hinders experimental throughput and ultimately slows down development and implementation. In single-walled carbon nanotube (CNT) synthesis, for instance, the poor yield obtained from conventional catalysts is a result of limited understanding of input-to-output correlations. Autonomous closed-loop experimentation combined with advances in machine learning (ML) is uniquely suited for high-throughput research. Among the ML algorithms available, Bayesian optimization (BO) is especially apt for exploration and optimization within such high-dimensional and complex parameter space. BO is an adaptive sequential design algorithm for finding the global optimum of a black-box objective function with the fewest possible measurements. Here, we demonstrate a promising application of BO in CNT synthesis as an efficient and robust algorithm which can (1) improve the growth rate of CNT in the BO-planner experiments over the seed experiments up to a factor 8; (2) rapidly improve its predictive power (or learning); (3) Consistently achieve good performance regardless of the number or origin of seed experiments; (4) exploit a high-dimensional, complex parameter space, and (5) achieve the former 4 tasks in just over 100 hundred experiments (~8 experimental hours) – a factor of 5× faster than our previously reported results.

An Improved Surrogate Based Optimization Method for Expensive Black-box Problems

For expensive black-box problems, surrogate modelling techniques are generally used to decrease the computational source. In this study, an improved surrogate based optimization (SBO) method is presented to solve the real-world engineering applications with expensive black-box objective responses. An optimized ensemble of surrogates combing three typical surrogate modelling techniques is adapted to efficiently predict the objective response. Meanwhile, the hierarchical design space reduction (HSR) strategy is employed for obtaining the smaller design subspace for improving the optimization efficiency. During the search, all test problems are considered as the real-world engineering applications whereas the actual global optima as well as the function characteristics are unknown in advance. The results show that the proposed method is superior in identifying the global optimum.

Surrogate Optimization of Computationally Expensive Black-Box Problems with Hidden Constraints

We introduce the algorithm SHEBO (surrogate optimization of problems with hidden constraints and expensive black-box objectives), an efficient optimization algorithm that employs surrogate models to solve computationally expensive black-box simulation optimization problems that have hidden constraints. Hidden constraints are encountered when the objective function evaluation does not return a value for a parameter vector. These constraints are often encountered in optimization problems in which the objective function is computed by a black-box simulation code. SHEBO uses a combination of local and global search strategies together with an evaluability prediction function and a dynamically adjusted evaluability threshold to iteratively select new sample points. We compare the performance of our algorithm with that of the mesh-based algorithms mesh adaptive direct search (MADS, NOMAD [nonlinear optimization by mesh adaptive direct search] implementation) and implicit filtering and SNOBFIT (stable noisy optimization by branch and fit), which assigns artificial function values to points that violate the hidden constraints. Our numerical experiments for a large set of test problems with 2–30 dimensions and a 31-dimensional real-world application problem arising in combustion simulation show that SHEBO is an efficient solver that outperforms the other methods for many test problems.