Derivative-based Optimization Methods Research Articles

Surrogate gradient methods for data-driven foundry energy consumption optimization

In many industrial applications, data-driven models are more and more commonly employed as an alternative to classical analytical descriptions or simulations. In particular, such models are often used to predict the outcome of an industrial process with respect to specific quality characteristics from both observed process parameters and control variables. A major step in proceeding from purely predictive to prescriptive analytics, i.e., towards leveraging data-driven models for process optimization, consists of, for given process parameters, determining control variable values such that the output quality improves according to the process model. This task naturally leads to a constrained optimization problem for data-driven prediction algorithms. In many cases, however, the best available models suffer from a lack of regularity: methods such as gradient boosting or random forests are generally non-differentiable and might even exhibit discontinuities. The optimization of these models would therefore require the use of derivative-free techniques. Here, we discuss the use of alternative, independently trained differentiable machine learning models as a surrogate during the optimization procedure. While these alternatives are generally less accurate representations of the actual process, the possibility of employing derivative-based optimization methods provides major advantages in terms of computational performance. Using classical benchmarks as well as a real-world dataset obtained from an industrial environment, we demonstrate that these advantages can outweigh the additional model error, especially in real-time applications.

A subset-selection-based derivative-free optimization algorithm for dynamic operation optimization in a steel-making process

ABSTRACT Steel-making production is a dynamic process that has the characteristics of high temperature and heat, and a complex reaction mechanism that causes the mechanism model possibly to be unavailable and the system to be a black box. In this article, a dynamic operation optimization (DOO) problem is refined from the basic oxygen furnace (BOF) steel-making process, and the system model is formulated by a data analytics method. This prevents to solve the optimization problem with derivative-based optimization methods. To circumvent these difficulties, a surrogate-model-based derivative-free optimization algorithm is proposed for solving the DOO problem. In order to establish the surrogate model with the least number of function evaluations, a subset selection strategy is designed to find a sparse structure for the optimization problem, based on which a set of simple bases is determined to establish the surrogate model. Moreover, this also reduces the scale of the parameter optimization problem. Numerical experiments on actual production data verify the applicability and effectiveness of the proposed method.

A Fast Method for Fuzzy Rules Learning with Derivative-Free Optimization by Formulating Independent Evaluations of Each Fuzzy Rule

A method is proposed for evaluating fuzzy rules independently of each other in fuzzy rules learning. The proposed method is named α-FUZZI-ES (α-weight-based fuzzy-rule independent evaluations) in this paper. In α-FUZZI-ES, the evaluation value of a fuzzy system is divided out among the fuzzy rules by using the compatibility degrees of the learning data. By the effective use of α-FUZZI-ES, a method for fast fuzzy rules learning is proposed. This is named α-FUZZI-ES learning (α-FUZZI-ES-based fuzzy rules learning) in this paper. α-FUZZI-ES learning is especially effective when evaluation functions are not differentiable and derivative-based optimization methods cannot be applied to fuzzy rules learning. α-FUZZI-ES learning makes it possible to optimize fuzzy rules independently of each other. This property reduces the dimensionality of the search space in finding the optimum fuzzy rules. Thereby, α-FUZZI-ES learning can attain fast convergence in fuzzy rules optimization. Moreover, α-FUZZI-ES learning can be efficiently performed with hardware in parallel to optimize fuzzy rules independently of each other. Numerical results show that α-FUZZI-ES learning is superior to the exemplary conventional scheme in terms of accuracy and convergence speed when the evaluation function is non-differentiable.

On Modelling of Data-Driven Monotone Zero-Order TSK Fuzzy Inference Systems using a System Identification Framework

A system identification-based framework is used to develop monotone fuzzy If-Then rules for formulating monotone zero-order Takagi-Sugeno-Kang (TSK) fuzzy inference systems (FISs) in this paper. Convex and normal trapezoidal and triangular fuzzy sets, together with a strong fuzzy partition strategy (either fixed or adaptive), is adopted. By coupling the strong fuzzy partition with a set of complete and monotone fuzzy If-Then rules, a monotone TSK FIS model can be guaranteed. We show that when a clean multiattribute monotone dataset is used, a system identification-based framework does not guarantee the production of monotone fuzzy If-Then rules, which leads to nonmonotone TSK FIS models. This is a new learning phenomenon that needs to be scrutinized when we design data-based monotone TSK FIS models. Two solutions are proposed: 1) a new monotone fuzzy rule relabeling-based method and 2) a constrained derivative-based optimization method. A new modeling framework with an adaptive fuzzy partition is evaluated. The results indicate that TSK FIS models with better accuracy (a lower sum square error) and a good degree of monotonicity (measured with a monotonicity test) are achieved. In short, the main contributions of this study are validation of the new learning phenomenon and introduction of useful methods for developing data-based monotone TSK FIS models.

A hybrid MILP and IPM approach for dynamic economic dispatch with valve-point effects

Dynamic economic dispatch with valve-point effects (DED-VPE) is a non-convex and non-differentiable optimization problem that is difficult to solve efficiently. In this paper, a hybrid approach combining mixed-integer linear programming (MILP) and the interior point method (IPM), abbreviated as MILP-IPM, is proposed to solve a DED-VPE problem in which complicated transmission loss is also included. Due to the non-differentiable nature of the DED-VPE problem, the classical derivative-based optimization methods can no longer be used. With the help of model reformulation, a differentiable non-linear programming (NLP) formulation that can be directly solved using the IPM is derived. However, if the DED-VPE problem is solved using the IPM in a single step, the optimization can easily become trapped in a poor local optimum due to its non-convex nature and multiple local minima. To obtain a better solution, an MILP method is required to solve the DED-VPE problem without transmission loss, yielding a good initial point for the IPM to improve the quality of the solution. Simulation results demonstrate the validity and effectiveness of the proposed MILP-IPM approach for solving the DED-VPE problem.

A differentiable optimization model for the management of single-species, even-aged stands

This paper presents a model for optimizing the management of single-species, even-aged stands. The model comprises a number of state variables, given by functions initially differentiable with respect to time, which cease to be due to the discontinuities that cause instant silvicultural treatments. Nevertheless, the proposed model maintains its differentiability with respect to the decision variables (timing, type, and intensity of each thinning, and rotation age). This allows for formulation of the problem of optimal management of this type of stands as a linearly constrained smooth optimization problem, which can be efficiently solved by any derivative-based optimization method. The effectiveness of this formulation is shown by using a sequential quadratic programming (SQP) algorithm to design the optimal management of Pinus pinaster Ait. in Asturias (northwestern Spain) from an economic perspective. These results are compared with those obtained using two methods that do not require derivatives. The SQP, as representative of gradient-type methods, proved robust and much more efficient than the derivative-free methods.

On the q-Weibull distribution for reliability applications: An adaptive hybrid artificial bee colony algorithm for parameter estimation

The q-Weibull model is based on the Tsallis non-extensive entropy [22] and is able to model various behaviors of the hazard rate function, including bathtub curves, by using a single set of parameters. Despite its flexibility, the q-Weibull has not been widely used in reliability applications partly because of the complicated parameters estimation. In this work, the parameters of the q-Weibull are estimated by the maximum likelihood (ML) method. Due to the intricate system of nonlinear equations, derivative-based optimization methods may fail to converge. Thus, the heuristic optimization method of artificial bee colony (ABC) is used instead. To deal with the slow convergence of ABC, it is proposed an adaptive hybrid ABC (AHABC) algorithm that dynamically combines Nelder-Mead simplex search method with ABC for the ML estimation of the q-Weibull parameters. Interval estimates for the q-Weibull parameters, including confidence intervals based on the ML asymptotic theory and on bootstrap methods, are also developed. The AHABC is validated via numerical experiments involving the q-Weibull ML for reliability applications and results show that it produces faster and more accurate convergence when compared to ABC and similar approaches. The estimation procedure is applied to real reliability failure data characterized by a bathtub-shaped hazard rate.

Shape optimization for contact problems based on isogeometric analysis

We consider the shape optimization for mechanical connectors. To avoid the gap between the representation in CAD systems and the finite element simulation used by mathematical optimization, we choose an isogeometric approach for the solution of the contact problem within the optimization method. This leads to a shape optimization problem governed by an elastic contact problem. We handle the contact conditions using the mortar method and solve the resulting contact problem with a semismooth Newton method. The optimization problem is nonconvex and nonsmooth due to the contact conditions. To reduce the number of simulations, we use a derivative based optimization method. With the adjoint approach the design derivatives can be calculated efficiently. The resulting optimization problem is solved with a modified Bundle Trust Region algorithm.

Applying Mathematical Optimization Methods to an ACT-R Instance-Based Learning Model.

Computational models of cognition provide an interface to connect advanced mathematical tools and methods to empirically supported theories of behavior in psychology, cognitive science, and neuroscience. In this article, we consider a computational model of instance-based learning, implemented in the ACT-R cognitive architecture. We propose an approach for obtaining mathematical reformulations of such cognitive models that improve their computational tractability. For the well-established Sugar Factory dynamic decision making task, we conduct a simulation study to analyze central model parameters. We show how mathematical optimization techniques can be applied to efficiently identify optimal parameter values with respect to different optimization goals. Beyond these methodological contributions, our analysis reveals the sensitivity of this particular task with respect to initial settings and yields new insights into how average human performance deviates from potential optimal performance. We conclude by discussing possible extensions of our approach as well as future steps towards applying more powerful derivative-based optimization methods.

Estimation of cardiac conductivities in ventricular tissue by a variational approach

The bidomain model is the current standard model to simulate cardiac potential propagation. The numerical solution of this system of partial differential equations strongly depends on the model parameters and in particular on the cardiac conductivities. Unfortunately, it is quite problematic to measure these parameters in vivo and even more so in clinical practice, resulting in no common agreement in the literature. In this paper we consider a variational data assimilation approach to estimating those parameters. We consider the parameters as control variables to minimize the mismatch between the computed and the measured potentials under the constraint of the bidomain system. The existence of a minimizer of the misfit function is proved with the phenomenological Rogers–McCulloch ionic model, that completes the bidomain system. We significantly improve the numerical approaches in the literature by resorting to a derivative-based optimization method with settlement of some challenges due to discontinuity. The improvement in computational efficiency is confirmed by a 2D test as a direct comparison with approaches in the literature. The core of our numerical results is in 3D, on both idealized and real geometries, with the minimal ionic model. We demonstrate the reliability and the stability of the conductivity estimation approach in the presence of noise and with an imperfect knowledge of other model parameters.

Stochastic Optimization Based 3D Dense Reconstruction from Multiple Views with High Accuracy and Completeness

This paper presents a stochastic optimization based 3D dense reconstruction from multiple views. Accuracy and completeness are two major measure indices for performance evaluation of various multi-view stereo (MVS) algorithms. First, the reconstruction accuracy is highly related to the stereo mismatches over the multiple views. Stereo mismatches occur in the image regions involving the lack of texture, depth discontinuity, or repeated texture patterns. Second, an insufficient number of views or occlusions between objects also lead to the difficulty in matching so that the reconstruction completeness degrades. In pursuit of high accuracy and completeness we present the appropriate techniques to solve the above problems in the reconstruction task. To deal with the various stereo mismatch problems we propose to apply adaptive matching functions and allow partial matching. We shall model the object to be reconstructed by a set of 3D oriented planar patches covering the visible object surface. The adopted multi-view reconstruction is formulated as a patch expansion process under a tree hierarchy. In order to find the optimal patches via multi-view stereo matching we shall employ a PSO (Particle Swarm Optimization) method for the sake of implementation simplicity and avoidance of possible local traps as found in the derivative based optimization methods. The success in the PSO method relies on imposing proper constraints on ranges of the patch parameters including the patch depth and patch normal vector which are involved in the PSO objective function (i.e., the stereo matching function). Furthermore, we use a varying patch size to obtain the reliable patches in the areas containing less texture, repeated texture pattern, or depth discontinuity. To secure a high reconstruction quality we advocate a patch priority queue to select the best patch during the patch expansion. All of the above mentioned techniques are also effective in the situations when the number of views is sparse or the camera baseline width is wide. The proposed method is tested on synthetic and real image data sets. The experimental results indicate that the proposed method is superior or comparable to the top ranked reconstruction methods reported in the public Middlebury MVS evaluation website.

Derivative-based hybrid heuristics for continuous-time simulation optimization

The topic of simulation–optimization has not been fundamentally tackled by many continuous-time modeling and simulation tools, yet. Common simulation-based optimization problems are usually coupled with standard optimization algorithms like any other simulation-free nonlinear optimization problems. While such couplings are usually based on many state-of-the-art software engineering concepts with a high-level user interface for flexible incorporation of simulation and optimization, the design of specialized optimization strategies targeting simulation-based objective functions is lacked within many simulation–optimization tools. In this work, new redefinition of Non Linear Programming (NLP) problems in the context of continuous-time simulation optimization is presented. Then, the modified optimization problems are efficiently tackled using derivative-based hybrid heuristics. In order to specify, illustrate and implement such heuristics, a new terminology is proposed. According to the proposed terminology, derivative-based hybrid strategies are implemented by hybridizing naive multistart derivative-based optimization methods with population-based metaheuristics. It is shown that the adoption of derivative-based optimization methods within hybrid optimization strategies significantly improves the solution quality of continuous-time simulation optimization problems.

Nonlinearly Constrained Optimization Using Heuristic Penalty Methods and Asynchronous Parallel Generating Set Search

Many optimization problems are characterized by expensive objective and/or constraint function evaluations paired with a lack of derivative information. Direct search methods such as generating set search (GSS) are well understood and efficient for derivative-free optimization of unconstrained and linearly constrained problems. This paper presents a study of heuristic algorithms that address the more difficult problem of general nonlinear constraints where derivatives for objective or constraint functions are unavailable. We focus on penalty methods that use GSS to solve a sequence of linearly constrained problems, numerically comparing different penalty functions. A classical choice for penalizing constraint violations is � 2 , the squared � 2 norm, which has advantages for derivative-based optimization methods. In our numerical tests, however, we show that exact penalty functions based on the � 1, � 2 ,a nd� ∞ norms converge to good approximate solutions more quickly and thus are attractive alternatives. Unfortunately, exact penalty functions are nondifferentiable and consequently degrade the final solution accuracy, so we also consider smoothed variants. Smoothed-exact penalty functions are attractive because they retain the differentiability of the original problem. Numerically, they are a compromise between exact and � 2 , i.e., they converge to a good solution somewhat quickly without sacrificing much solution accuracy. Moreover, the smoothing is parameterized and can potentially be adjusted to balance the two considerations. Since our

Non-derivative design centering algorithm using trust region optimization and variance reduction

Fluctuations in manufactured integrated circuit parameters may dramatically reduce the parametric yield. Yield maximization can be formulated as an unconstrained optimization problem in nominal parameter values, which is known as design centering. The high expense of yield evaluations, the absence of any gradient information, and the presence of some numerical noise obstruct the use of the traditional derivative-based optimization methods. In this article, a novel design centering algorithm is presented, which consists of a non-derivative unconstrained optimizer coupled with a variance reduction estimator. The used optimizer combines a trust region mechanism with quadratic interpolation and provides efficient use of yield evaluations. The stratified sampling technique is used to develop a lower variance yield estimator that reduces the number of circuit simulations required to reach a desired accuracy level. Numerical and practical circuit examples are used to demonstrate the efficiency of the proposed algorithm with respect to other methods in the same field.

A smooth shear strength function for steel flexural members

The strength of web plates in shear may be governed by one of three types of behavior: yielding, inelastic buckling, or elastic buckling. A separate function is needed to define the strength for each of these failure modes. Consequently, the shear strength of flexural members is defined by a piecewise continuous function that varies with the ratio of web height to thickness. The shear strength function is continuous, but its first derivative is not. Optimization of steel members with such nonsmooth constraint functions is limited because most algorithms assume the objective and constraint functions are smooth. In order to improve this situation, an alternative function is sought for the inelastic (transition) region such that the first derivative is continuous or nearly so. Analytical and finite element models as well as experimental data are used to predict the inelastic buckling behavior. This data is used to fit a polynomial of p -th order ( p > 1 ) for use in design. The value of p which results in a smooth curve may not agree well with the finite element results, so depending on how much nonsmoothness can be tolerated, a nearly smooth but conservative curve may be most suitable for optimization applications. Such a curve is appropriate for design using optimization methods since it models the inelastic buckling behavior of webs in shear and does not violate the assumptions of derivative-based optimization methods.

Heuristic optimization for consistent initialization of DAEs

The optimization approach seems promising for the consistent initialization of DAEs for it can be applied to the resolution of over-determined or ill-conditioned algebraic consistency systems. Given the DAE system and all the additional constraints (feasible or not) that the user may impose, it must be found a set of initial conditions as close as possible to a consistent one. In other words, the deviation from the equations and constraints must be minimized in some norm. The choice of the optimization method is crucial for the successful implementation of the code. Derivative-based optimization methods and stochastic optimization methods are quite different techniques, with different drawbacks and potentialities. When combined, these strategies generally lead to very efficient search and optimization tools. There has been a growing interest in heuristic optimization methods since the emergence of massively parallel computers. However, the impact of such techniques on the dynamic simulation of chemical engineering systems — particularly in the initialization problem — has not been sufficiently studied. In several classes of important problems traditional optimization techniques may fail, due to either high computational cost or inherent limitations of gradient-type methods. A preliminary DAE initialization code using the optimization approach and a heuristic optimization algorithm, under development at PEQ/COPPE/UFRJ, is presented in this work. The code has been tested in some chemical engineering applications. Simple models frequently adopted in the DAE literature were used to provide a kind of benchmark for the initialization problem.