Black-Box Optimization Benchmarking Research Articles

Adapting the population size in CMA-ES using nearest-better clustering method for multimodal optimization

The covariance matrix adaptation evolution strategy (CMA-ES) is one of the advanced algorithms for solving continuous single-objective optimization. In this algorithm, the population size is an important parameter which represents the number of candidate points generated in each iteration. A larger population size allows for better exploration of the search space and is typically required for handling multimodal functions, while a smaller population size facilitates quicker convergence. Adapting the population size in this algorithm raises a question about how to increase/decrease/keep it in an efficient way to solve specific problems. In this paper, we propose a novel approach for this purpose based on the information of niches observed during the evolution process. Firstly, the nearest-better clustering (NBC) technique is employed to detect the number of niches in each iteration, then the information will be used to adapt the population size accordingly. Additionally, a quasi-Newton method, which serves as a local search, can be incorporated into the algorithm in order to speed it up. We evaluate the effectiveness of our method through numerical simulations on multimodal test functions, the black-box optimization benchmarking (BBOB) noiseless testbed, and the CEC 2014 and CEC 2017 benchmark testing suites.

A Novel Improved Whale Optimization Algorithm for Global Optimization and Engineering Applications

The Whale Optimization Algorithm (WOA) is a swarm intelligence algorithm based on natural heuristics, which has gained considerable attention from researchers and engineers. However, WOA still has some limitations, including limited global search efficiency and a slow convergence rate. To address these issues, this paper presents an improved whale optimization algorithm with multiple strategies, called Dynamic Gain-Sharing Whale Optimization Algorithm (DGSWOA). Specifically, a Sine–Tent–Cosine map is first adopted to more effectively initialize the population, ensuring a more uniform distribution of individuals across the search space. Then, a gaining–sharing knowledge based algorithm is used to enhance global search capability and avoid falling into a local optimum. Finally, to increase the diversity of solutions, Dynamic Opposition-Based Learning is incorporated for population updating. The effectiveness of our approach is evaluated through comparative experiments on blackbox optimization benchmarking and two engineering application problems. The experimental results suggest that the proposed method is competitive in terms of solution quality and convergence speed in most cases.

Generating Cheap Representative Functions for Expensive Automotive Crashworthiness Optimization

Solving real-world engineering optimization problems, such as automotive crashworthiness optimization, is extremely challenging, because the problem characteristics are oftentimes not well understood. Furthermore, typical hyperparameter optimization approaches that require a large function evaluation budget are computationally hindered, if the function evaluation is expensive, e.g., requires finite element (FE) simulation runs. In this paper, we propose an approach to characterize real-world expensive black-box optimization problems using the exploratory landscape analysis (ELA). Based on these landscape characteristics, we can identify test functions that are fast-to-evaluate and representative for hyperparameter optimization purposes. Focusing on 20 problem instances from automotive crashworthiness optimization, our results reveal that these 20 crashworthiness problems exhibit landscape features different from classical optimization benchmark test suites, such as the widely-used black-box optimization benchmarking (BBOB) problem set. In fact, these 20 problem instances belong to problem classes that are distinct from the BBOB test functions based on the clustering results. Further analysis indicates that, as far as the ELA features concern, they are most similar to problem classes of tree-based test functions. By analyzing the performance of two optimization algorithms with different hyperparameters, namely the covariance matrix adaptation evolutionary strategy (CMA-ES) and Bayesian optimization (BO), we show that the tree-based test functions are indeed representative in terms of predicting the algorithm performances. Following this, such scalable and fast-to-evaluate tree-based test functions have promising potential for automated design of an optimization algorithm for specific real-world problem classes.

OPTION: OPTImization Algorithm Benchmarking ONtology

Many optimization algorithm benchmarking platforms allow users to share their experimental data to promote reproducible and reusable research. However, different platforms use different data models and formats, which drastically complicates the identification of relevant datasets, their interpretation, and their interoperability. Therefore, a semantically rich, ontology-based, machine-readable data model that can be used by different platforms is highly desirable. In this paper, we report on the development of such an ontology, which we call OPTION (OPTImization algorithm benchmarking ONtology). Our ontology provides the vocabulary needed for semantic annotation of the core entities involved in the benchmarking process, such as algorithms, problems, and evaluation measures. It also provides means for automatic data integration, improved interoperability, and powerful querying capabilities, thereby increasing the value of the benchmarking data. We demonstrate the utility of OPTION, by annotating and querying a corpus of benchmark performance data from the BBOB collection of the COCO framework and from the Yet Another Black-Box Optimization Benchmark (YABBOB) family of the Nevergrad environment. In addition, we integrate features of the BBOB functional performance landscape into the OPTION knowledge base using publicly available datasets with exploratory landscape analysis. Finally, we integrate the OPTION knowledge base into the IOHprofiler environment and provide users with the ability to perform meta-analysis of performance data.

Optimizing With Low Budgets: A Comparison On the Black-Box Optimization Benchmarking Suite and OpenAI Gym

Optimizing With Low Budgets: A Comparison On the Black-Box Optimization Benchmarking Suite and OpenAI Gym

Anytime Performance Assessment in Blackbox Optimization Benchmarking

We present concepts and recipes for the anytime performance assessment when benchmarking optimization algorithms in a blackbox scenario. We consider runtime—oftentimes measured in the number of blackbox evaluations needed to reach a target quality—to be a universally measurable cost for solving a problem. Starting from the graph that depicts the solution quality versus runtime, we argue that runtime is the only performance measure with a generic, meaningful, and quantitative interpretation. Hence, our assessment is solely based on runtime measurements. We discuss proper choices for solution quality indicators in single- and multi-objective optimization, as well as in the presence of noise and constraints. We also discuss the choice of the target values, budget-based targets, and the aggregation of runtimes by using simulated restarts, averages, and empirical cumulative distributions which generalize convergence graphs of single runs. The presented performance assessment is to a large extent implemented in the comparing continuous optimizers (COCO) platform freely available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/numbbo/coco</uri> .

Benchmarking Feature-Based Algorithm Selection Systems for Black-Box Numerical Optimization

Feature-based algorithm selection aims to automatically find the best one from a portfolio of optimization algorithms on an unseen problem based on its landscape features. Feature-based algorithm selection has recently received attention in the research field of black-box numerical optimization. However, there is still room for the analysis of algorithm selection for black-box optimization. Most previous studies have focused only on whether an algorithm selection system can outperform the single-best solver (SBS) in a portfolio. In addition, a benchmarking methodology for algorithm selection systems has not been well investigated in the literature. In this context, this article analyzes algorithm selection systems on the 24 noiseless black-box optimization benchmarking functions. First, we demonstrate that the first successful performance measure is more reliable than the expected runtime measure for benchmarking algorithm selection systems. Then, we examine the influence of randomness on the performance of algorithm selection systems. We also show that the performance of algorithm selection systems can be significantly improved by using sequential least squares programming as a presolver. We point out that the difficulty of outperforming the SBS depends on algorithm portfolios, cross-validation methods, and dimensions. Finally, we demonstrate that the effectiveness of algorithm portfolios depends on various factors. These findings provide fundamental insights for algorithm selection for black-box optimization.

Influencer buddy optimization: Algorithm and its application to electricity load and price forecasting problem

Swarm-based algorithms are widely accepted in different fields of engineering as they have proven themselves effective in solving real-world optimization problems. According to the “No-Free-Lunch” theorem, there is no such algorithm that can perform well in all optimization problems. This paper proposes a novel swarm-based optimization algorithm, influencer buddy optimization (IBO), that mimics human behavior in social environments. This algorithm exploits the fact that humans are influenced by a group of individuals rather than a single person. However, every individual can select a buddy from this group based on their preferences. The mathematical representation of this social human behavior is shown in this paper. The algorithm is tested on twenty-one standard benchmark functions, and twenty-four noiseless black-box optimization benchmarking (BBOB) functions to validate its effectiveness over fourteen optimization algorithms. Further, two hybrid models are proposed by combining IBO with an artificial neural network (ANN) and recurrent neural network (RNN) to solve electricity load and price forecasting problems. The results of hybrid models are compared with nine recent and popular optimization algorithms. The simulation result of hybrid models shows that ANN-IBO and RNN-IBO generate the least forecasting error in their respective classes of network architectures.

Noisy Optimization by Evolution Strategies With Online Population Size Learning

Optimization modeling of real-world application problems usually involves noise from various sources. Noisy optimization imposes challenges to optimization methods since the objective values can be different for multiple evaluations. In this article, we propose a novel online population size learning (OPL) technique of evolution strategies for handling noisy optimization problems. By re-evaluating a fraction of the candidates, we measure the strength of noise level of the re-evaluated candidate solutions and adapt the population size according to the noise level. The proposed OPL combines the advantages of both explicit averaging by re-evaluations and the implicit averaging by large population size and overcomes their limitations. We incorporate it with the covariance matrix adaptation evolution strategy (CMA-ES) and obtain OPL-CMA-ES. Compared with the existing noise handling technique, the proposed OPL is much simpler in both concepts and computation. We conduct comprehensive experiments to evaluate the algorithm’s performance on standard problems with Gaussian noise. We further evaluate the performance of OPL-CMA-ES on the black-box optimization benchmarks (BBOBs) noisy testbed, which is a standard platform for comparing black-box optimization algorithms, compared with the state-of-the-art noise-handling algorithms. The experimental results show that OPL-CMA-ES achieves remarkable performance and outperforms the compared variants.

Using Well-Understood Single-Objective Functions in Multiobjective Black-Box Optimization Test Suites.

Several test function suites are being used for numerical benchmarking of multiobjective optimization algorithms. While they have some desirable properties, such as well-understood Pareto sets and Pareto fronts of various shapes, most of the currently used functions possess characteristics that are arguably underrepresented in real-world problems such as separability, optima located exactly at the boundary constraints, and the existence of variables that solely control the distance between a solution and the Pareto front. Via the alternative construction of combining existing single-objective problems from the literature, we describe the bbob-biobj test suite with 55 bi-objective functions in continuous domain, and its extended version with 92 bi-objective functions (bbob-biobj-ext). Both test suites have been implemented in the COCO platform for black-box optimization benchmarking and various visualizations of the test functions are shown to reveal their properties. Besides providing details on the construction of these problems and presenting their (known) properties, this article also aims at giving the rationale behind our approach in terms of groups of functions with similar properties, objective space normalization, and problem instances. The latter allows us to easily compare the performance of deterministic and stochastic solvers, which is an often overlooked issue in benchmarking.

A Comparative Study of Common Nature-Inspired Algorithms for Continuous Function Optimization.

Over previous decades, many nature-inspired optimization algorithms (NIOAs) have been proposed and applied due to their importance and significance. Some survey studies have also been made to investigate NIOAs and their variants and applications. However, these comparative studies mainly focus on one single NIOA, and there lacks a comprehensive comparative and contrastive study of the existing NIOAs. To fill this gap, we spent a great effort to conduct this comprehensive survey. In this survey, more than 120 meta-heuristic algorithms have been collected and, among them, the most popular and common 11 NIOAs are selected. Their accuracy, stability, efficiency and parameter sensitivity are evaluated based on the 30 black-box optimization benchmarking (BBOB) functions. Furthermore, we apply the Friedman test and Nemenyi test to analyze the performance of the compared NIOAs. In this survey, we provide a unified formal description of the 11 NIOAs in order to compare their similarities and differences in depth and a systematic summarization of the challenging problems and research directions for the whole NIOAs field. This comparative study attempts to provide a broader perspective and meaningful enlightenment to understand NIOAs.

Comparison of sampling methods for algorithm configuration problem: A case for tuning differential ant-stigmergy (DASA) algorithm parameters

Metaheuristic and heuristic methods, which are effective tools for solving difficult computational problems, have many design parameters, and selecting their values affects their performance. A good selection of the parameter values can increase their performance. In this study, we compared eight different sampling methods, used for tuning the five control parameters of the Differential Ant-Stigmergy Algorithm (DASA). Although most of the proposed parameter tuning methods in the literature use sampling methods to initialize or select candidate parameter vectors, we used them as individual tuning methods in this study. This is the first study to use Latin Hypercube Hammersley Sampling (LHHS) for algorithm configuration problems. In the first part of this study, the performance of a large set of DASA parameter settings, obtained using different sampling methods, was evaluated on the Sphere function optimization problem with respect to two dimensions of this function. In the second part of the experiment, we compared the performance of these differently tuned versions of DASA on the first seven noiseless benchmark test functions of the Black-Box Optimization Benchmarking 2010 workshop. The results of our research demonstrated that the best parameter vector obtained with the LHHS method had a better performance than the default parameter value of the DASA and other best parameter values obtained with other methods (it has the best-ranking amongst the all compared method). Furthermore, according to our results, LHHS can produce promising results when used for the initialization of other state-of-art algorithm configuration methods.

An Adaptive Fuzzy Chicken Swarm Optimization Algorithm

The chicken swarm optimization (CSO) algorithm is a new swarm intelligence optimization (SIO) algorithm and has been widely used in many engineering domains. However, there are two apparent problems with the CSO algorithm, i.e., slow convergence speed and difficult to achieve global optimal solutions. Aiming at attacking these two problems of CSO, in this paper, we propose an adaptive fuzzy chicken swarm optimization (FCSO) algorithm. The proposed FCSO uses the fuzzy system to adaptively adjust the number of chickens and random factors of the CSO algorithm and achieves an optimal balance of exploitation and exploration capabilities of the algorithm. We integrate the cosine function into the FCSO to compute the position update of roosters and improve the convergence speed. We compare the FCSO with eight commonly used, state-of-the-art SIO algorithms in terms of performance in both low- and high-dimensional spaces. We also verify the FCSO algorithm with the nonparametric statistical Friedman test. The results of the experiments on the 30 black-box optimization benchmarking (BBOB) functions demonstrate that our FCSO outperforms the other SIO algorithms in both convergence speed and optimization accuracy. In order to further test the applicability of the FCSO algorithm, we apply it to four typical engineering problems with constraints on the optimization processes. The results show that the FCSO achieves better optimization accuracy over the standard CSO algorithm.

Differential Evolution Optimal Parameters Tuning with Artificial Neural Network

Differential evolution (DE) is a simple and efficient population-based stochastic algorithm for solving global numerical optimization problems. DE largely depends on algorithm parameter values and search strategy. Knowledge on how to tune the best values of these parameters is scarce. This paper aims to present a consistent methodology for tuning optimal parameters. At the heart of the methodology is the use of an artificial neural network (ANN) that learns to draw links between the algorithm performance and parameter values. To do so, first, a data-set is generated and normalized, then the ANN approach is performed, and finally, the best parameter values are extracted. The proposed method is evaluated on a set of 24 test problems from the Black-Box Optimization Benchmarking (BBOB) benchmark. Experimental results show that three distinct cases may arise with the application of this method. For each case, specifications about the procedure to follow are given. Finally, a comparison with four tuning rules is performed in order to verify and validate the proposed method’s performance. This study provides a thorough insight into optimal parameter tuning, which may be of great use for users.

A SIGEVO impact award for a paper arising from the COCO platform

The paper Comparing Results of 31 Algorithms from the Black-Box Optimization Benchmarking BBOB-2009 received the 2020 SIGEVO Impact Award for ten-year impact, which was announce at the ACM GECCO 2020 conference. The work compares the performance of 31 algorithms that had been benchmarked with the Comparing Continuous Optimizer platform (COCO) for the Black-Box-Optimization Benchmarking (BBOB) GECCO workshop in 2009.

GOPS: efficient RBF surrogate global optimization algorithm with high dimensions and many parallel processors including application to multimodal water quality PDE model calibration

This paper describes a new parallel global surrogate-based algorithm Global Optimization in Parallel with Surrogate (GOPS) for the minimization of continuous black-box objective functions that might have multiple local minima, are expensive to compute, and have no derivative information available. The task of picking P new evaluation points for P processors in each iteration is addressed by sampling around multiple center points at which the objective function has been previously evaluated. The GOPS algorithm improves on earlier algorithms by (a) new center points are selected based on bivariate non-dominated sorting of previously evaluated points with additional constraints to ensure the objective value is below a target percentile and (b) as iterations increase, the number of centers decreases, and the number of evaluation points per center increases. These strategies and the hyperparameters controlling them significantly improve GOPS’s parallel performance on high dimensional problems in comparison to other global optimization algorithms, especially with a larger number of processors. GOPS is tested with up to 128 processors in parallel on 14 synthetic black-box optimization benchmarking test problems (in 10, 21, and 40 dimensions) and one 21-dimensional parameter estimation problem for an expensive real-world nonlinear lake water quality model with partial differential equations that takes 22 min for each objective function evaluation. GOPS numerically significantly outperforms (especially on high dimensional problems and with larger numbers of processors) the earlier algorithms SOP and PSD-MADS-VNS (and these two algorithms have outperformed other algorithms in prior publications).

Understanding the problem space in single-objective numerical optimization using exploratory landscape analysis

In benchmarking theory, creating a comprehensive and uniformly distributed set of problems is a crucial first step to designing a good benchmark. However, this step is also one of the hardest, as it can be difficult to determine how to evaluate the quality of the chosen problem set.In this article, we evaluate if the field of exploratory landscape analysis can be used to develop a generalized method of visualizing a set of arbitrary optimization functions. We present a method for visually determining the distribution of problems within a benchmark set using exploratory landscape analysis combined with clustering and t-sne visualization, and evaluate and explain the visualization this methodology produces.The proposed method is evaluated on a set of benchmark problems taken from two well known state-of-the-art real-parameter single objective optimization benchmarks: the CEC Special Sessions and Competitions on Real-Parameter Single Objective optimization, and the GECCO Black-Box Optimization Benchmark workshops.The main goal of this paper is to present an analysis of how exploratory landscape analysis can be used to visualize a benchmark problem set. We show that this method can provide a clear visualization of a benchmark problem set and shows the similarities of the problems in it by placing similar problems visually close together. We also show that the problem sets of the above benchmarks have a somewhat distinct set of problems that do not overlap.In addition, by applying feature selection approaches we show that a number of landscape features provided by state-of-the-art exploratory landscape analysis libraries are redundant and that a large amount of them are not invariant to simple transforms like scaling and shifting, at least when analyzing these two datasets.

Comparing optimization algorithms for conventional and freeform optical design.

Five search algorithms from the literature of black-box optimization were implemented and applied to optical design problems. These algorithms are the Particle Swarm Optimization, Gravity Search Algorithm, Cuckoo Search, Covariance Matrix Adaptation Evolution Strategy and Nelder&Mead Simplex search. The performance of these search algorithms' implementations was assessed using the BBOB2009 (Black Box Optimization Benchmark) benchmark suite. These algorithms were compared in the context of two optical case studies, one with conventional rotationally symmetric optics and one with freeform optics. A comparison was performed against a commercial optical design software. Finally we provided a simple restart scheme applicable in the workflow of an optical designer. To the best of our knowledge, this is the first in-depth quantitative comparison of optimization algorithms for optical design.

Mirrored Orthogonal Sampling for Covariance Matrix Adaptation Evolution Strategies.

Generating more evenly distributed samples in high dimensional search spaces is the major purpose of the recently proposed mirrored sampling technique for evolution strategies. The diversity of the mutation samples is enlarged and the convergence rate is therefore improved by the mirrored sampling. Motivated by the mirrored sampling technique, this article introduces a new derandomized sampling technique called mirrored orthogonal sampling. The performance of this new technique is both theoretically analyzed and empirically studied on the sphere function. In particular, the mirrored orthogonal sampling technique is applied to the well-known Covariance Matrix Adaptation Evolution Strategy (CMA-ES). The resulting algorithm is experimentally tested on the well-known Black-Box Optimization Benchmark (BBOB). By comparing the results from the benchmark, mirrored orthogonal sampling is found to outperform both the standard CMA-ES and its variant using mirrored sampling.

A Comparative Study of PSO and CMA-ES Algorithms on Black-box Optimization Benchmarks

Numerous practical engineering applications can be formulated as non-convex, non-smooth, multi-modal and ill-conditioned optimization problems. Classical, deterministic algorithms require an enormous computational effort, which tends to fail as the problem size and its complexity increase, which is often the case. On the other hand, stochastic, biologically-inspired techniques, designed for global optimum calculation, frequently prove successful when applied to real life computational problems. While the area of bio-inspired algorithms (BIAs) is still relatively young, it is undergoing continuous, rapid development. Selection and tuning of the appropriate optimization solver for a particular task can be challenging and requires expert knowledge of the methods to be considered. Comparing the performance of viable candidates against a defined test bed environment can help in solving such dilemmas. This paper presents the benchmark results of two biologically inspired algorithms: covariance matrix adaptation evolution strategy (CMA-ES) and two variants of particle swarm optimization (PSO). COCO (COmparing Continuous Optimizers) – a platform for systematic and sound comparisons of real-parameter global optimization solvers was used to evaluate the performance of CMA-ES and PSO methods. Particular attention was paid to the efficiency and scalability of both techniques.