Automatic Algorithm Configuration Research Articles

Extending the "Open-Closed Principle" to Automated Algorithm Configuration.

Metaheuristics are an effective and diverse class of optimization algorithms: a means of obtaining solutions of acceptable quality for otherwise intractable problems. The selection, construction, and configuration of a metaheuristic for a given problem has historically been a manually intensive process based on experience, experimentation, and reasoning by metaphor. More recently, there has been interest in automating the process of algorithm configuration. In this article, we identify shared state as an inhibitor of progress for such automation. To solve this problem, we introduce the Automated Open-Closed Principle (AOCP), which stipulates design requirements for unintrusive reuse of algorithm frameworks and automated assembly of algorithms from an extensible palette of components. We demonstrate how the AOCP enables a greater degree of automation than previously possible via an example implementation.

Latin Hypercube Designs with Branching and Nested Factors for Initialization of Automatic Algorithm Configuration.

The configuration of algorithms is a laborious and difficult process. Thus, it is advisable to automate this task by using appropriate automatic configuration methods. The method is among the most widely used in the literature. By default, initializes its search process via uniform sampling of algorithm configurations. Although better initialization methods exist in the literature, the mixed-variable (numerical and categorical) nature of typical parameter spaces and the presence of conditional parameters make most of the methods not applicable in practice. Here, we present an improved initialization method that overcomes these limitations by employing concepts from the design and analysis of computer experiments with branching and nested factors. Our results show that this initialization method is not only better, in some scenarios, than the uniform sampling used by the current version of , but also better than other initialization methods present in other automatic configuration methods.

Automatic Configuration of Multi-Objective Local Search Algorithms for Permutation Problems.

Automatic algorithm configuration (AAC) is becoming a key ingredient in the design of high-performance solvers for challenging optimisation problems. However, most existing work on AAC deals with configuration procedures that optimise a single performance metric of a given, single-objective algorithm. Of course, these configurators can also be used to optimise the performance of multi-objective algorithms, as measured by a single performance indicator. In this work, we demonstrate that better results can be obtained by using a native, multi-objective algorithm configuration procedure. Specifically, we compare three AAC approaches: one considering only the hypervolume indicator, a second optimising the weighted sum of hypervolume and spread, and a third that simultaneously optimises these complementary indicators, using a genuinely multi-objective approach. We assess these approaches by applying them to a highly-parametric local search framework for two widely studied multi-objective optimisation problems, the bi-objective permutation flowshop and travelling salesman problems. Our results show that multi-objective algorithms are indeed best configured using a multi-objective configurator.

Deep optimization for spectrum repacking

Over 13 months in 2016--17 the U.S. Federal Communications Commission conducted an "incentive auction" to repurpose radio spectrum from broadcast television to wireless internet. In the end, the auction yielded $19.8 bn, $10.05 bn of which was paid to 175 broadcasters for voluntarily relinquishing their licenses across 14 Ultra High Frequency (UHF) channels. Stations that continued broadcasting were assigned potentially new channels to fit as densely as possible into the channels that remained. The government netted more than $7 bn (used to pay down the national debt) after covering costs (including retuning). A crucial element of the auction design was the construction of a solver, dubbed SAT-based Feasibility Checker (SATFC), that determined whether sets of stations could be "repacked" in this way; it needed to run every time a station was given a price quote. This paper describes the process by which we built SATFC. We adopted an approach we dub "deep optimization," taking a data-driven, highly parametric, and computationally intensive approach to solver design. More specifically, to build SATFC we designed software that could pair both complete and local-search SAT-encoded feasibility checking with a wide range of domain-specific techniques, such as constraint graph decomposition and novel caching mechanisms that allow for reuse of partial solutions from related, solved problems. We then used automatic algorithm configuration techniques to construct a portfolio of 8 complementary algorithms to be run in parallel, aiming to achieve good performance on instances that arose in proprietary auction simulations. To evaluate the impact of our solver in this paper, we built an open-source reverse auction simulator. We found that within the short time budget required in practice, SATFC solved more than 95% of the problems it encountered. Furthermore, the incentive auction paired with SATFC produced nearly optimal allocations in a restricted setting and substantially outperformed other alternatives at national scale.

A Large-Scale Experimental Evaluation of High-Performing Multi- and Many-Objective Evolutionary Algorithms.

Research on multi-objective evolutionary algorithms (MOEAs) has produced over the past decades a large number of algorithms and a rich literature on performance assessment tools to evaluate and compare them. Yet, newly proposed MOEAs are typically compared against very few, often a decade older MOEAs. One reason for this apparent contradiction is the lack of a common baseline for comparison, with each subsequent study often devising its own experimental scenario, slightly different from other studies. As a result, the state of the art in MOEAs is a disputed topic. This article reports a systematic, comprehensive evaluation of a large number of MOEAs that covers a wide range of experimental scenarios. A novelty of this study is the separation between the higher-level algorithmic components related to multi-objective optimization (MO), which characterize each particular MOEA, and the underlying parameters-such as evolutionary operators, population size, etc.-whose configuration may be tuned for each scenario. Instead of relying on a common or "default" parameter configuration that may be low-performing for particular MOEAs or scenarios and unintentionally biased, we tune the parameters of each MOEA for each scenario using automatic algorithm configuration methods. Our results confirm some of the assumed knowledge in the field, while at the same time they provide new insights on the relative performance of MOEAs for many-objective problems. For example, under certain conditions, indicator-based MOEAs are more competitive for such problems than previously assumed. We also analyze problem-specific features affecting performance, the agreement between performance metrics, and the improvement of tuned configurations over the default configurations used in the literature. Finally, the data produced is made publicly available to motivate further analysis and a baseline for future comparisons.

Exploring variable neighborhood search for automatic algorithm configuration

Methods for automatic algorithm configuration integrate some search mechanism for generating candidate algorithm configurations and mechanisms for handling the stochasticity of the algorithm configuration problem. One popular algorithm configurator is ParamILS, which searches the configuration space using an iterated local search algorithm. In our research, we explore variable neighborhood search mechanisms as an alternative for the one-exchange neighborhood that is searched in the local search phase of ParamILS. In this article, we explore a reduced variable neighborhood search for automatic configuration. Our experimental results are promising and indicate directions for extending our work.

ABC-X: a generalized, automatically configurable artificial bee colony framework

The artificial bee colony (ABC) algorithm is a popular metaheuristic that was originally conceived for tackling continuous function optimization tasks. Over the last decade, a large number of variants of ABC have been proposed, making it by now a well-studied swarm intelligence algorithm. Typically, in a paper on algorithmic variants of ABC algorithms, one or at most two of its algorithmic components are modified. Possible changes include variations on the search equations, the selection of candidate solutions to be explored, or the adoption of features from other algorithmic techniques. In this article, we propose to follow a different direction and to build a generalized ABC algorithm, which we call ABC-X. ABC-X collects algorithmic components available from known ABC algorithms into a common algorithm framework that allows not only to instantiate known ABC variants but, more importantly, also many ABC algorithm variants that have never been explored before in the literature. Automatic algorithm configuration techniques can generate from this template new ABC variants that perform better than known ABC algorithms, even when their numerical parameters are fine-tuned using the same automatic configuration process.

MIP neighborhood synthesis through semantic feature extraction and automatic algorithm configuration

The definition of a “good” neighborhood structure on the solution space is a key step when designing several types of heuristics for Mixed Integer Programming (MIP). Typically, in order to achieve efficiency in the search, the neighborhood structures need to be tailored not only to the specific problem but also to the peculiar distribution of the instances to be solved (reference instance population). Nowadays, this is done by human experts through a time-consuming process comprising: (a) problem analysis, (b) literature scouting and (c) experimentation. In this paper, we illustrate an Automatic Neighborhood Design algorithm that mimics steps (a) and (c). Firstly, the procedure extracts some semantic features from a MIP compact model. Secondly, these features are used to derive automatically some neighborhood design mechanisms. Finally, the “proper mix” of such mechanisms is sought through an automatic configuration phase performed on a training set representative of the reference instance population. When assessed on four well-known combinatorial optimization problems, our automatically-generated neighborhoods outperform state-of-the-art model-based neighborhoods with respect to both scalability and solution quality.

Automatic instantiation of a Variable Neighborhood Descent from a Mixed Integer Programming model

In this paper we describe the automatic instantiation of a Variable Neighborhood Descent procedure from a Mixed Integer Programming model. We extend a recent approach in which a single neighborhood structure is automatically designed from a Mixed Integer Programming model using a combination of automatic extraction of semantic features and automatic algorithm configuration. Computational results on four well-known combinatorial optimization problems show improvements over both a previous model-derived Variable Neighborhood Descent procedure and the approach with a single automatically-designed neighborhood structure.

Automatic Component-Wise Design of Multiobjective Evolutionary Algorithms

Multiobjective evolutionary algorithms (MOEAs) are typically proposed, studied, and applied as monolithic blocks with a few numerical parameters that need to be set. Few works have studied how the algorithmic components of these evolutionary algorithms can be classified and combined to produce new algorithmic designs. The motivation for studies of this latter type stem from the development of flexible software frameworks and the usage of automatic algorithm configuration methods to find novel algorithm designs. In this paper, we propose anMOEA template and a new conceptual view of its components that surpasses existing frameworks in both number of algorithms that can be instantiated from the template and flexibility to produce novel algorithmic designs. We empirically demonstrate the flexibility of our proposed framework by automatically designing MOEAs for continuous and combinatorial optimization problems. The automatically designed algorithms are often able to outperform six traditional MOEAs from the literature, even after tuning their numerical parameters.

Automatic construction of parallel portfolios via algorithm configuration

Since 2004, increases in computational power described by Moore's law have substantially been realized in the form of additional cores rather than through faster clock speeds. To make effective use of modern hardware when solving hard computational problems, it is therefore necessary to employ parallel solution strategies. In this work, we demonstrate how effective parallel solvers for propositional satisfiability (SAT), one of the most widely studied NP-complete problems, can be produced automatically from any existing sequential, highly parametric SAT solver. Our Automatic Construction of Parallel Portfolios (ACPP) approach uses an automatic algorithm configuration procedure to identify a set of configurations that perform well when executed in parallel. Applied to two prominent SAT solvers, Lingeling and clasp, our ACPP procedure identified 8-core solvers that significantly outperformed their sequential counterparts on a diverse set of instances from the application and hard combinatorial category of the 2012 SAT Challenge. We further extended our ACPP approach to produce parallel portfolio solvers consisting of several different solvers by combining their configuration spaces. Applied to the component solvers of the 2012 SAT Challenge gold medal winning SAT Solver pfolioUZK, our ACPP procedures produced a significantly better-performing parallel SAT solver.

Bayesian Optimization in a Billion Dimensions via Random Embeddings

Bayesian optimization techniques have been successfully applied to robotics, planning, sensor placement, recommendation, advertising, intelligent user interfaces and automatic algorithm configuration. Despite these successes, the approach is restricted to problems of moderate dimension, and several workshops on Bayesian optimization have identified its scaling to high-dimensions as one of the holy grails of the field. In this paper, we introduce a novel random embedding idea to attack this problem. The resulting Random EMbedding Bayesian Optimization (REMBO) algorithm is very simple, has important invariance properties, and applies to domains with both categorical and continuous variables. We present a thorough theoretical analysis of REMBO. Empirical results confirm that REMBO can effectively solve problems with billions of dimensions, provided the intrinsic dimensionality is low. They also show that REMBO achieves state-of-the-art performance in optimizing the 47 discrete parameters of a popular mixed integer linear programming solver.

The irace package: Iterated racing for automatic algorithm configuration

Modern optimization algorithms typically require the setting of a large number of parameters to optimize their performance. The immediate goal of automatic algorithm configuration is to find, automatically, the best parameter settings of an optimizer. Ultimately, automatic algorithm configuration has the potential to lead to new design paradigms for optimization software. The irace package is a software package that implements a number of automatic configuration procedures. In particular, it offers iterated racing procedures, which have been used successfully to automatically configure various state-of-the-art algorithms. The iterated racing procedures implemented in irace include the iterated F-race algorithm and several extensions and improvements over it. In this paper, we describe the rationale underlying the iterated racing procedures and introduce a number of recent extensions. Among these, we introduce a restart mechanism to avoid premature convergence, the use of truncated sampling distributions to handle correctly parameter bounds, and an elitist racing procedure for ensuring that the best configurations returned are also those evaluated in the highest number of training instances. We experimentally evaluate the most recent version of irace and demonstrate with a number of example applications the use and potential of irace, in particular, and automatic algorithm configuration, in general.

Modeling the Throughput of the Linux-Based Agile-SD Transmission Control Protocol

Agile-SD is one of the latest versions of loss-based congestion control algorithm (CCA), which has been proposed to improve the total performance of transmission control protocol (TCP) over high-speed and short-distance networks. It has introduced a new mechanism, called agility factor mechanism, which shortens the epoch time to reduce the sensitivity to packet losses and in turn to increase the average throughput. Agile-SD has only been tested via simulation; however, it has not been mathematically proven or evaluated. The contribution of this paper is twofold. First, a new mathematical model for the throughput of NewReno and Agile-SD is proposed. This model is designed using the well-known Markov chains to validate the correctness of Agile-SD and to show the impact of buffer size, multiplicative decrease factor, and maximum limit of agility factor ({\lambda}max ) on the total performance. Second, an automated algorithm configuration and parameter tuning (AACPT) technique is employed to optimize and automate the configuration of {\lambda}max . Furthermore, the numerical results for both NewReno and Agile-SD are compared with the simulation results, in which the validity of the proposed model is confirmed. Moreover, the output of the AACPT is exploited to formulate a new equation, which calculates the optimal {\lambda}max from a given \b{eta} in order to conserve the standard interface of the TCP. This equation increases the scalability of Agile-SD and improves its total performance.

SATenstein: Automatically building local search SAT solvers from components

Designing high-performance solvers for computationally hard problems is a difficult and often time-consuming task. Although such design problems are traditionally solved by the application of human expertise, we argue instead for the use of automatic methods. In this work, we consider the design of stochastic local search (SLS) solvers for the propositional satisfiability problem (SAT). We first introduce a generalized, highly parameterized solver framework, dubbed SATenstein, that includes components drawn from or inspired by existing high-performance SLS algorithms for SAT. The parameters of SATenstein determine which components are selected and how these components behave; they allow SATenstein to instantiate many high-performance solvers previously proposed in the literature, along with trillions of novel solver strategies. We used an automated algorithm configuration procedure to find instantiations of SATenstein that perform well on several well-known, challenging distributions of SAT instances. Our experiments show that SATenstein solvers achieved dramatic performance improvements as compared to the previous state of the art in SLS algorithms; for many benchmark distributions, our new solvers also significantly outperformed all automatically tuned variants of previous state-of-the-art algorithms.

AutoFolio: An Automatically Configured Algorithm Selector

Algorithm selection (AS) techniques -- which involve choosing from a set of algorithms the one expected to solve a given problem instance most efficiently -- have substantially improved the state of the art in solving many prominent AI problems, such as SAT, CSP, ASP, MAXSAT and QBF. Although several AS procedures have been introduced, not too surprisingly, none of them dominates all others across all AS scenarios. Furthermore, these procedures have parameters whose optimal values vary across AS scenarios. This holds specifically for the machine learning techniques that form the core of current AS procedures, and for their hyperparameters. Therefore, to successfully apply AS to new problems, algorithms and benchmark sets, two questions need to be answered: (i) how to select an AS approach and (ii) how to set its parameters effectively. We address both of these problems simultaneously by using automated algorithm configuration. Specifically, we demonstrate that we can automatically configure claspfolio 2, which implements a large variety of different AS approaches and their respective parameters in a single, highly-parameterized algorithm framework. Our approach, dubbed AutoFolio, allows researchers and practitioners across a broad range of applications to exploit the combined power of many different AS methods. We demonstrate AutoFolio can significantly improve the performance of claspfolio 2 on 8 out of the 13 scenarios from the Algorithm Selection Library, leads to new state-of-the-art algorithm selectors for 7 of these scenarios, and matches state-of-the-art performance (statistically) on all other scenarios. Compared to the best single algorithm for each AS scenario, AutoFolio achieves average speedup factors between 1.3 and 15.4.

Hyperopt: a Python library for model selection and hyperparameter optimization

Sequential model-based optimization (also known as Bayesian optimization) is one of the most efficient methods (per function evaluation) of function minimization. This efficiency makes it appropriate for optimizing the hyperparameters of machine learning algorithms that are slow to train. The Hyperopt library provides algorithms and parallelization infrastructure for performing hyperparameter optimization (model selection) in Python. This paper presents an introductory tutorial on the usage of the Hyperopt library, including the description of search spaces, minimization (in serial and parallel), and the analysis of the results collected in the course of minimization. This paper also gives an overview of Hyperopt-Sklearn, a software project that provides automatic algorithm configuration of the Scikit-learn machine learning library. Following Auto-Weka, we take the view that the choice of classifier and even the choice of preprocessing module can be taken together to represent a single large hyperparameter optimization problem. We use Hyperopt to define a search space that encompasses many standard components (e.g. SVM, RF, KNN, PCA, TFIDF) and common patterns of composing them together. We demonstrate, using search algorithms in Hyperopt and standard benchmarking data sets (MNIST, 20-newsgroups, convex shapes), that searching this space is practical and effective. In particular, we improve on best-known scores for the model space for both MNIST and convex shapes. The paper closes with some discussion of ongoing and future work.

Population-based heuristic algorithms for continuous and mixed discrete-continuous optimization problems

Continuous optimization problems are optimization problems where all variableshave a domain that typically is a subset of the real numbers; mixed discrete-continuousoptimization problems have additionally other types of variables, sothat some variables are continuous and others are on an ordinal or categoricalscale. Continuous and mixed discrete-continuous problems have a wide rangeof applications in disciplines such as computer science, mechanical or electricalengineering, economics and bioinformatics. These problems are also often hard tosolve due to their inherent difficulties such as a large number of variables, manylocal optima or other factors making problems hard. Therefore, in this thesis ourfocus is on the design, engineering and configuration of high-performing heuristicoptimization algorithms.We tackle continuous and mixed discrete-continuous optimization problemswith two classes of population-based heuristic algorithms, ant colony optimization(ACO) algorithms and evolution strategies. In a nutshell, the main contributionsof this thesis are that (i) we advance the design and engineering of ACO algorithms to algorithms that are competitive or superior to recent state-of-the-artalgorithms for continuous and mixed discrete-continuous optimization problems,(ii) we improve upon a specific state-of-the-art evolution strategy, the covariancematrix adaptation evolution strategy (CMA-ES), and (iii) we extend CMA-ES totackle mixed discrete-continuous optimization problems.More in detail, we propose a unified ant colony optimization (ACO) frameworkfor continuous optimization (UACOR). This framework synthesizes algorithmiccomponents of two ACO algorithms that have been proposed in the literatureand an incremental ACO algorithm with local search for continuous optimization,which we have proposed during my doctoral research. The design of UACORallows the usage of automatic algorithm configuration techniques to automaticallyderive new, high-performing ACO algorithms for continuous optimization. We alsopropose iCMAES-ILS, a hybrid algorithm that loosely couples IPOP-CMA-ES, aCMA-ES variant that uses a restart schema coupled with an increasing populationsize, and a new iterated local search (ILS) algorithm for continuous optimization.The hybrid algorithm consists of an initial competition phase, in which IPOP-CMA-ES and the ILS algorithm compete for further deployment during a secondphase. A cooperative aspect of the hybrid algorithm is implemented in the formof some limited information exchange from IPOP-CMA-ES to the ILS algorithmduring the initial phase. Experimental studies on recent benchmark functionssuites show that UACOR and iCMAES-ILS are competitive or superior to otherstate-of-the-art algorithms.To tackle mixed discrete-continuous optimization problems, we extend ACOMV and propose CESMV, an ant colony optimization algorithm and a covariance matrix adaptation evolution strategy, respectively. In ACOMV and CESMV , the decision variables of an optimization problem can be declared as continuous, ordinal, or categorical, which allows the algorithm to treat them adequately. ACOMV andCESMV include three solution generation mechanisms: a continuous optimizationmechanism, a continuous relaxation mechanism for ordinal variables, and a categorical optimization mechanism for categorical variables. Together, these mechanisms allow ACOMV and CESMV to tackle mixed variable optimization problems.We also propose a set of artificial, mixed-variable benchmark functions, which cansimulate discrete variables as ordered or categorical. We use them to automatically tune ACOMV and CESMV's parameters and benchmark their performance.Finally we test ACOMV and CESMV on various real-world continuous and mixed-variable engineering optimization problems. Comparisons with results from theliterature demonstrate the effectiveness and robustness of ACOMV and CESMVon mixed-variable optimization problems.Apart from these main contributions, during my doctoral research I have accomplished a number of additional contributions, which concern (i) a note on thebound constraints handling for the CEC'05 benchmark set, (ii) computational results for an automatically tuned IPOP-CMA-ES on the CEC'05 benchmark set and(iii) a study of artificial bee colonies for continuous optimization. These additionalcontributions are to be found in the appendix to this thesis.

Automatic Configuration of Sequential Planning Portfolios

Sequential planning portfolios exploit the complementary strengths of different planners. Similarly, automated algorithm configuration tools can customize parameterized planning algorithms for a given type of tasks. Although some work has been done towards combining portfolios and algorithm configuration, the problem of automatically generating a sequential planning portfolio from a parameterized planner for a given type of tasks is still largely unsolved. Here, we present Cedalion, a conceptually simple approach for this problem that greedily searches for the pair of parameter configuration and runtime which, when appended to the current portfolio, maximizes portfolio improvement per additional runtime spent. We show theoretically that Cedalion yields portfolios provably within a constant factor of optimal for the training set distribution. We evaluate Cedalion empirically by applying it to construct sequential planning portfolios based on component planners from the highly parameterized Fast Downward (FD) framework. Results for a broad range of planning settings demonstrate that -- without any knowledge of planning or FD -- Cedalion constructs sequential FD portfolios that rival, and in some cases substantially outperform, manually-built FD portfolios.

Grammar-based generation of stochastic local search heuristics through automatic algorithm configuration tools

Several grammar-based genetic programming algorithms have been proposed in the literature to automatically generate heuristics for hard optimization problems. These approaches specify the algorithmic building blocks and the way in which they can be combined in a grammar; the best heuristic for the problem being tackled is found by an evolutionary algorithm that searches in the algorithm design space defined by the grammar.In this work, we propose a novel representation of the grammar by a sequence of categorical, integer, and real-valued parameters. We then use a tool for automatic algorithm configuration to search for the best algorithm for the problem at hand. Our experimental evaluation on the one-dimensional bin packing problem and the permutation flowshop problem with weighted tardiness objective shows that the proposed approach produces better algorithms than grammatical evolution, a well-established variant of grammar-based genetic programming. The reasons behind such improvement lie both in the representation proposed and in the method used to search the algorithm design space.