Derivative-free Optimization Algorithm Research Articles

Global linear convergence of evolution strategies with recombination on scaling-invariant functions

Evolution Strategies (ESs) are stochastic derivative-free optimization algorithms whose most prominent representative, the CMA-ES algorithm, is widely used to solve difficult numerical optimization problems. We provide the first rigorous investigation of the linear convergence of step-size adaptive ESs involving a population and recombination, two ingredients crucially important in practice to be robust to local irregularities or multimodality. We investigate the convergence of step-size adaptive ESs with weighted recombination on composites of strictly increasing functions with continuously differentiable scaling-invariant functions with a global optimum. This function class includes functions with non-convex sublevel sets and discontinuous functions. We prove the existence of a constant r such that the logarithm of the distance to the optimum divided by the number of iterations converges to r. The constant is given as an expectation with respect to the stationary distribution of a Markov chain—its sign allows to infer linear convergence or divergence of the ES and is found numerically. Our main condition for convergence is the increase of the expected log step-size on linear functions. In contrast to previous results, our condition is equivalent to the almost sure geometric divergence of the step-size on linear functions.

Comparison of parallel optimization algorithms on computationally expensive groundwater remediation designs

Contaminated groundwater resources threaten human health and destroy ecosystems in many areas worldwide. Groundwater remediation is crucial for environmental recovery; however, it can be cost prohibitive. Planning a cost-effective remediation design can take a long time, as it may involve the evaluation of many management decisions, and the corresponding simulation models are computationally demanding. Parallel optimization can facilitate much faster management decisions for cost-effective groundwater remediation design using complex pollutant transport models. However, the efficiency of different parallel optimization algorithms varies depending on both the search strategy and parallelism. In this paper, we show the performance of a parallel surrogate-based optimization algorithm called parallel stochastic radial basis function (p-SRBF), which has not been previously used on contaminant remediation problems. The two case problems involve two superfund sites (i.e., the Umatilla Aquifer and Blaine Aquifer), and one objective evaluation takes 5 and 30 min for the two problems, respectively. Exceptional speedup (superlinear) is achieved with 4 to 16 cores, and excellent speedup is achieved using up to 64 cores, obtaining a good solution at 80 % efficiency. We compare our p-SRBF with three different parallel derivative-free optimization algorithms, including genetic algorithm, mesh adaptive direct search, and asynchronous parallel pattern search optimization, in terms of objective quality, computational reduction and robust behavior across multiple trials. p-SRBF outperforms the other algorithms, as it finds the best solution in both the Umatilla and Blaine cases and reduces the computational budget by at least 50 % in both problems. Additionally, statistical comparisons show that the p-SRBF results are better than those of the alternative algorithms at the 5 % significant level. This study enriches theoretical real-world groundwater remediation methods. The results demonstrate that p-SRBF is promising for environmental management problems that involve computationally expensive models.

A subset-selection-based derivative-free optimization algorithm for dynamic operation optimization in a steel-making process

ABSTRACT Steel-making production is a dynamic process that has the characteristics of high temperature and heat, and a complex reaction mechanism that causes the mechanism model possibly to be unavailable and the system to be a black box. In this article, a dynamic operation optimization (DOO) problem is refined from the basic oxygen furnace (BOF) steel-making process, and the system model is formulated by a data analytics method. This prevents to solve the optimization problem with derivative-based optimization methods. To circumvent these difficulties, a surrogate-model-based derivative-free optimization algorithm is proposed for solving the DOO problem. In order to establish the surrogate model with the least number of function evaluations, a subset selection strategy is designed to find a sparse structure for the optimization problem, based on which a set of simple bases is determined to establish the surrogate model. Moreover, this also reduces the scale of the parameter optimization problem. Numerical experiments on actual production data verify the applicability and effectiveness of the proposed method.

A Fractional-Order On-Line Self Optimizing Control Framework and a Benchmark Control System Accelerated Using Fractional-Order Stochasticity

This paper presents a design and evaluation of a fractional-order self optimizing control (FOSOC) architecture for process control. It is based on a real-time derivative-free optimization layer that adjusts the parameters of a discrete-time fractional-order proportional integral (FOPI) controller according to an economic cost function. A simulation benchmark is designed to assess the performance of the FOSOC controller based on a first order plus dead time system. Similarly, an acceleration mechanism is proposed for the fractional-order self optimizing control framework employing fractional-order Gaussian noise with long-range dependence given by the Hurst exponent. The obtained results show that the FOSOC controller can improve the system closed-loop response under different operating conditions and reduce the convergence time of the real-time derivative-free optimization algorithm by using fractional-order stochasticity.

Finite element model updating through derivative-free optimization algorithm

Finite element (FE) model of updating is the process of calibrating model parameters to improve the accuracy of numerical prediction. This goal is usually achieved by solving an optimization problem with the objective function measuring the misfit between simulated responses and experimental data. The sensitivity method is the predominate class of algorithms to FE model updating problems, and has opened a wide range of applications. However, this method often suffers from large errors due to linearization. To address this challenge, this paper proposes to utilize the unscented Kalman inversion (UKI) method to solve the FE model updating problems in a derivative-free manner. As an iterative optimization method, the UKI determines the new iterate using function values at a set of sample points rather than the derivative information. Implementation details, such as handling constraints during the optimization process, are presented in the paper. To validate the proposed UKI, model updating of a pedestrian bridge is conducted using the simulated and experimental data. Both validation examples show that the UKI is a competitive method for solving FE model updating problems.

Dynamic characteristics and adaptive design methods of enclosed courtyard: A case study of a single-story courtyard dwelling in China

In this study, the dynamic performance of an enclosed courtyard with a covered roof in a cold IIA region of China was explored, and the microclimate regulation and energy performance of its component, the courtyard adjustable ventilated roof (CsAVR), were emphasized. As a theoretical basis for thermal context exploration, adaptive thermal comfort content was introduced, and mean thermal sensation vote ( MTS ) was adopted as one of the evaluation metrics. The thermal environment was monitored, and subsequently, energy models and numerical models were constructed in DesignBuilder and the finite element program COMSOL, respectively, which were both validated by the measured data. The comparative experiments were applied to the energy model to validate the capabilities of CsAVR on thermal regulation and energy savings, while a morphological optimization procedure based on derivative-free optimization algorithms was implemented in COMSOL to obtain the optimal set of design parameters for CsAVR satisfying thermal comfort. The comparative experiments revealed that the CsAVR reduced the operative temperature in the courtyard and adjacent space by 3.9 °C and 1.6 °C, respectively, during the warm period; approximately 47.4% (courtyard) and 28.6% (adjacent space) of the occupied hours were extended to the comfort range during the cooling season; and approximately 8.2% of the energy was saved throughout the year. With the input of meteorological parameters, dwelling geometry parameters, and thermal adaptation objective functions, an optimization program with an embedded COBYLA algorithm was created to rapidly obtain a combination of morphological control parameters that approximates optimal thermal comfort. • Compare the performance of covered and uncovered courtyards. • The thermal and microclimatic functions of the courtyard are both considered. • A thermal intervention design that responds to the thermal comfort and behavior of occupants. • The MTS -restricted morphological optimization program is available.

High-dimensional normalized data profiles for testing derivative-free optimization algorithms.

This article provides a new tool for examining the efficiency and robustness of derivative-free optimization algorithms based on high-dimensional normalized data profiles that test a variety of performance metrics. Unlike the traditional data profiles that examine a single dimension, the proposed data profiles require several dimensions in order to analyze the relative performance of different optimization solutions. To design a use case, we utilize five sequences (solvers) of trigonometric simplex designs that extract different features of non-isometric reflections, as an example to show how various metrics (dimensions) are essential to provide a comprehensive evaluation about a particular solver relative to others. In addition, each designed sequence can rotate the starting simplex through an angle to designate the direction of the simplex. This type of features extraction is applied to each sequence of the triangular simplexes to determine a global minimum for a mathematical problem. To allocate an optimal sequence of trigonometric simplex designs, a linear model is used with the proposed data profiles to examine the convergence rate of the five simplexes. Furthermore, we compare the proposed five simplexes to an optimized version of the Nelder–Mead algorithm known as the Genetic Nelder–Mead algorithm. The experimental results demonstrate that the proposed data profiles lead to a better examination of the reliability and robustness for the considered solvers from a more comprehensive perspective than the existing data profiles. Finally, the high-dimensional data profiles reveal that the proposed solvers outperform the genetic solvers for all accuracy tests.

MR-double-zero – Proof-of-concept for a framework to autonomously discover MRI contrasts

PurposeA framework for supervised design of MR sequences for any given target contrast is proposed, based on fully automatic acquisition and reconstruction of MR data on a real MR scanner. The proposed method does not require any modeling of MR physics and thus allows even unknown contrast mechanisms to be addressed. MethodsA derivative-free optimization algorithm is set up to repeatedly update and execute a parametrized sequence on the MR scanner to acquire data. In each iteration, the acquired data are mapped to a given target contrast by linear regression. ResultsIt is shown that with the proposed framework it is possible to find an MR sequence that yields a predefined target contrast. In the present case, as a proof-of principle, a sequence mapping absolute creatine concentration, which cannot be extracted from T1 or T2-weighted scans directly, is discovered. The sequence was designed in a comparatively short time and with no human interaction. ConclusionsNew MR contrasts for mapping a given target can be discovered by derivative-free optimization of parametrized sequences that are directly executed on a real MRI scanner. This is demonstrated by ‘re-discovery’ of a chemical exchange weighted sequence. The proposed method is considered to be a paradigm shift towards autonomous, model-free and target-driven sequence design.

Using first-order information in direct multisearch for multiobjective optimization

Derivatives are an important tool for single-objective optimization. In fact, it is commonly accepted that derivative-based methods present a better performance than derivative-free optimization approaches. In this work, we will show that the same does not always apply to multiobjective derivative-based optimization, when the goal is to compute an approximation to the complete Pareto front of a given problem. The competitiveness of direct multisearch (DMS), a robust and efficient derivative-free optimization algorithm, will be stated for derivative-based multiobjective optimization (MOO) problems, by comparison with MOSQP, a state-of-art derivative-based MOO solver. We will then assess the potential enrichment of adding first-order information to the DMS framework. Derivatives will be used to prune the positive spanning sets considered at the poll step of the algorithm. The role of ascent directions, that conform to the geometry of the nearby feasible region, will then be highlighted.

Pressure capability analysis of magnetic powder seals and pole tooth design by multiparameter optimization

Ferrofluid seals have been an increasingly popular sealing technique with unique advantages like zero leakage, long lifetime and low friction torque. But ferrofluid seals are incapable of specific working conditions, due to intolerance of extreme temperatures and limited pressure capability per stage. Recently, a novel sealing method using magnetic powders has been put forward to solve the problems. However, studies on the seal performance and theory of design are still blank. Here for the first time, we give a theoretical analysis of pressure capability of magnetic powder seals. The formula of pressure capability is derived, and a determinant coefficient is put forward. For sealing structure design, four derivative-free optimization algorithms are used to optimize geometric parameters of pole teeth. Finally, a sealing test bench was established, and the experimental data were compared with the simulation results. This research provides a basic theoretical and experimental guidance for magnetic powder seals.

Multigroup strategy for well control optimization

Practical production optimization often involves determining a large number of well control settings such that an objective function is maximized or minimized. When derivative-free population-based optimization algorithms are applied for these high-dimensional problems, inefficient performance or slow convergence can be observed. In this work, we develop a multigroup optimization strategy to mitigate these issues. The first step of the method entails the ranking of decision variables based on their statistical significance, as observed during a short preprocessing optimization run. Variable groupings are then established. Group-by-group optimization, in which only a subset of decision variables is considered at each stage, is then conducted. As a final step, additional (limited) iterations with the full set of variables are performed. Particle swarm optimization (PSO), a global stochastic search method, is applied in this study, though the multigroup treatment is compatible with a range of algorithms. Detailed performance comparisons between multigroup strategies and standard optimization are presented for a model based on the 3D Brugge case. The problem involves 64 wells and 320 decision variables. The multigroup optimization framework includes several algorithmic parameters, and the use of different settings for these is shown to provide a range of promising procedures. Specifically, one such multigroup treatment requires 72% of the flow simulations of standard optimization, but it provides a final net present value (NPV) that is 2.6% higher than that from standard optimization. Another procedure provides essentially the same NPV as standard optimization but with only 40% of the flow simulations. The multigroup framework developed in this study can be extended to treat other challenging high-dimensional problems such as field development optimization.

Exploiting Problem Structure in Derivative Free Optimization

A structured version of derivative-free random pattern search optimization algorithms is introduced, which is able to exploit coordinate partially separable structure (typically associated with sparsity) often present in unconstrained and bound-constrained optimization problems. This technique improves performance by orders of magnitude and makes it possible to solve large problems that otherwise are totally intractable by other derivative-free methods. A library of interpolation-based modelling tools is also described, which can be associated with the structured or unstructured versions of the initial pattern search algorithm. The use of the library further enhances performance, especially when associated with structure. The significant gains in performance associated with these two techniques are illustrated using a new freely-available release of the Brute Force Optimizer (BFO) package firstly introduced in [Porcelli and Toint 2017 ], which incorporates them. An interesting conclusion of the numerical results presented is that providing global structural information on a problem can result in significantly less evaluations of the objective function than attempting to building local Taylor-like models.

Use of Static Surrogates in Hyperparameter Optimization

Optimizing the hyperparameters and architecture of a neural network is a long yet necessary phase in most applications. This consuming process can benefit from strategies designed to discard low-quality configurations and quickly focus on more promising candidates. This work aims at enhancing HyperNOMAD, a library that adapts a direct search derivative-free optimization algorithm to tune both the architecture and the training of a neural network simultaneously. Two static surrogates are developed to trigger an early stopping during the configuration evaluation and strategically rank a pool of candidates. These additions to HyperNOMAD are shown to reduce its resource consumption by orders of magnitude without harming the quality of the proposed solutions.

Fuzzy-tree-constructed data-efficient modelling methodology for volumetric efficiency of dedicated hybrid engines

The accurate characterization of volumetric efficiency is essential for modern combustion engines to achieve better performance, lower emissions, and reduced fuel consumption. To minimize experimental effort on sample collection and maintain high-precision volumetric efficiency characterization, this paper proposes a new methodology of fuzzy-tree-constructed data-efficient modelling to precisely quantify the air mass flow through the engine. Differing from conventional data-driven modelling, this methodology introduces a hierarchical fuzzy inference tree (HFIT) with three original topologies that accommodates simplicity by combining several low-dimensional fuzzy inference systems. Driven by two derivative-free optimization algorithms, a two-step tuning process is introduced to speed up the convergence process when traversing HFIT parameters. A Gaussian distributed resampling technique is developed to screen a small number of samples with diverse engine operations to maintain sample diversity. The experimental dataset is obtained from steady-state tests carried out in a BYD 1.5L gasoline engine specially made for a hybrid powertrain. The results demonstrate that the proposed fuzzy-tree-constructed data-efficient modelling methodology performs with superior learning efficiency on volumetric efficiency characterization than those of a fuzzy inference system, a neural network, or an adaptive neuro-fuzzy inference system. Even when dataset split ratio downs to 0.2, the relative mean absolute error can be restricted to 3.18% with the help of Gaussian distributed resampling technique.

Optimization of composite cylinder shell via a data-driven intelligent optimization algorithm

While composite material provides huge flexibility for design, the design optimization of composite structure is time consuming with low efficiency. This work combines finite element analysis for composite cylinder shell with a data-driven intelligent optimization algorithm (Bayesian optimization algorithm) and is aimed at maximizing eigenvalue buckling load. Through minimizing number of iterations as a derivative-free global optimization algorithm, Bayesian optimization is versatile and can be further applied to design advanced composite structure with more complicated scenarios, such as complex geometries and load conditions.

An efficient derivative-free optimization algorithm inspired by avian life-saving manoeuvres

A novel meta-heuristic algorithm is introduced that is inspired by aerial escaping strategies of a bird to avoid being captured by the hunter. The escaping bird chooses its life-saving maneuver based on the mass and velocity of it and of the predator bird. These flights are simulated in the developed algorithm as movements of artificial search agents. Hunting flight of the artificial predator, plays the role of search intensification; while the escaping prey explores new places in the space by flying in the opposite direction or making a turn. The method adaptively tunes the flight lengths in each pair of the prey/predator agents to avoid use of extra control parameters. It takes advantage of swarm intelligence and opposition-based learning. Twelve test functions as well as seven engineering problems in discrete, continuous and mixed discrete-continuous types are successfully solved by the proposed Escaping Bird Search; exhibiting its high convergence rate and competitive performance in comparison with the other treated meta-heuristics.

Data-driven optimization for process systems engineering applications

Most optimization problems in engineering can be formulated as ‘expensive’ black box problems whose solutions are limited by the number of function evaluations. Frequently, engineers develop accurate models of physical systems that are differentiable and/or cheap to evaluate. These models can be solved efficiently, and the solution transferred to the real system. In the absence of gradient information or cheap-to-evaluate models, one must resort to efficient optimization routines that rely only on function evaluations. Creating a model can itself be considered part of the expensive black box optimization process. In this work, we investigate how perceived state-of-the-art derivative-free optimization (DFO) algorithms address different instances of these problems in process engineering. On the algorithms side, we benchmark both model-based and direct-search DFO algorithms. On the problems side, the comparisons are made on one mathematical optimization problem and five chemical engineering applications: model-based design of experiments, flowsheet optimization, real-time optimization, self-optimizing reactions, and controller tuning. Various challenges are considered such as constraint satisfaction, uncertainty, problem dimension and evaluation cost. This work bridges the gap between the derivative-free optimization and process systems literature by providing insight into the efficiency of data-driven optimization algorithms in the process systems domain to advance the digitalization of the chemical and process industries.

Derivative-free optimization adversarial attacks for graph convolutional networks.

In recent years, graph convolutional networks (GCNs) have emerged rapidly due to their excellent performance in graph data processing. However, recent researches show that GCNs are vulnerable to adversarial attacks. An attacker can maliciously modify edges or nodes of the graph to mislead the model’s classification of the target nodes, or even cause a degradation of the model’s overall classification performance. In this paper, we first propose a black-box adversarial attack framework based on derivative-free optimization (DFO) to generate graph adversarial examples without using gradient and apply advanced DFO algorithms conveniently. Second, we implement a direct attack algorithm (DFDA) using the Nevergrad library based on the framework. Additionally, we overcome the problem of large search space by redesigning the perturbation vector using constraint size. Finally, we conducted a series of experiments on different datasets and parameters. The results show that DFDA outperforms Nettack in most cases, and it can achieve an average attack success rate of more than 95% on the Cora dataset when perturbing at most eight edges. This demonstrates that our framework can fully exploit the potential of DFO methods in node classification adversarial attacks.

Randomized Simplicial Hessian Update

Recently, a derivative-free optimization algorithm was proposed that utilizes a minimum Frobenius norm (MFN) Hessian update for estimating the second derivative information, which in turn is used for accelerating the search. The proposed update formula relies only on computed function values and is a closed-form expression for a special case of a more general approach first published by Powell. This paper analyzes the convergence of the proposed update formula under the assumption that the points from Rn where the function value is known are random. The analysis assumes that the N+2 points used by the update formula are obtained by adding N+1 vectors to a central point. The vectors are obtained by transforming a prototype set of N+1 vectors with a random orthogonal matrix from the Haar measure. The prototype set must positively span a N≤n dimensional subspace. Because the update is random by nature we can estimate a lower bound on the expected improvement of the approximate Hessian. This lower bound was derived for a special case of the proposed update by Leventhal and Lewis. We generalize their result and show that the amount of improvement greatly depends on N as well as the choice of the vectors in the prototype set. The obtained result is then used for analyzing the performance of the update based on various commonly used prototype sets. One of the results obtained by this analysis states that a regular n-simplex is a bad choice for a prototype set because it does not guarantee any improvement of the approximate Hessian.

A new biologically inspired global optimization algorithm based on firebug reproductive swarming behaviour

A new biologically inspired derivative-free global optimization algorithm called Firebug Swarm Optimization (FSO) inspired by reproductive swarming behaviour of Firebugs (Pyrrhocoris apterus) is proposed. The search for fit reproductive partners by individual bugs in a swarm of Firebugs can be viewed naturally as a search for optimal solutions in a search space. This work proposes a mathematical model for five different Firebug behaviours most relevant to optimization and uses these behaviours as the basis of a new global optimization algorithm. Performance of the FSO algorithm is compared with 17 popular heuristic algorithms on the Congress of Evolutionary Computation 2013 (CEC 2013) benchmark suite that contains high dimensional multimodal as well as shifted and rotated functions. Statistical analysis based on Wilcoxon Rank-Sum Test indicates that the proposed FSO algorithm outperforms 17 popular state-of-the-art heuristic global optimization algorithms like Guided Sparks Fireworks Algorithm (GFWA), Dynamic Learning PSO (DNLPSO), and Artificial Bee Colony Bollinger Bands (ABCBB) on the CEC 2013 benchmark.