Derivative-based Optimization Research Articles

Overview of Trust-region Methods

The trust-region technique is an optimization algorithm for solving multidimensional nonlinear optimization complications. It is a class of derivative-based optimization approaches that relies on information about the gradient of the objective function and possibly Hessians. The main idea behind the Trust-region method is to remodel the objective function around the present iteration Solution of the inner subproblem. The confidence region is the region around the present iteration where the model is expected to be accurate. At each iteration, the trust region technique uses a quadratic model to estimate the objective function locally (Yuan, 2019). This model is based on the current iteration and the information about the gradient and Hessian. The confidence region limits the distance the algorithm can take from the current iteration to find the next one. The size of the trust region is adjusted dynamically based on the arrangement between the model and the actual performance values.

A novel approach for optimizing the natural gas liquefaction process

The conversion of natural gas into liquefied natural gas (LNG) requires substantial energy consumption. This study proposes a deterministic mathematical model to find the optimal operation conditions for an LNG plant to minimize the energy consumption for turbomachinery and the total thermal conductance of the heat exchangers. General Algebraic Modeling System (GAMS) linked to a Dynamic Link Library was used to calculate the thermodynamic properties of the working fluids. A derivative-based optimization algorithm is used. Results indicate that the novel optimization approach allows the satisfactory management of the model nonlinearities associated, for example, with the bilinear terms involved in the energy balances and the mathematical functions used to calculate the thermodynamic properties. A preprocessing phase for initializing process variables is developed to facilitate model convergence. In comparison to an optimal design reported in the literature, which was obtained by integrating a well-established evolutionary optimization approach with the Aspen HYSYS simulator, the results indicated that the net electrical power could be reduced by up to 10% when the derivative-based optimization algorithm is used. The proposed deterministic approach, consisting of a mathematical model, an initialization phase, and an optimization algorithm, can help process engineers overcome the challenges associated with LNG process optimization.

Simulation-optimization framework for the digital design of pharmaceutical processes using Pyomo and PharmaPy.

The problem of performing model-based process design and optimization in the pharmaceutical industry is an important and challenging one both computationally and in choice of solution implementation. In this work, a framework is presented to directly utilize a process simulator via callbacks during derivative-based optimization. The framework allows users with little experience in translating mechanistic ODEs and PDEs to robust, fully discretized algebraic formulations, required for executing simultaneous equation-oriented optimization, to obtain mathematically guaranteed optima at a competitive solution time when compared with existing derivative-free and derivative-based frameworks. The effectiveness of the framework in accuracy of optimal solution as well as computational efficiency is analyzed on on two case studies: (i) an integrated 2-unit reaction synthesis train used for the synthesis of an anti-cancer active pharmaceutical ingredient, and (ii) a more complex flowsheet representing a common synthesis-purification-isolation train of a pharmaceutical manufacturing processes.

A subset-selection-based derivative-free optimization algorithm for dynamic operation optimization in a steel-making process

ABSTRACT Steel-making production is a dynamic process that has the characteristics of high temperature and heat, and a complex reaction mechanism that causes the mechanism model possibly to be unavailable and the system to be a black box. In this article, a dynamic operation optimization (DOO) problem is refined from the basic oxygen furnace (BOF) steel-making process, and the system model is formulated by a data analytics method. This prevents to solve the optimization problem with derivative-based optimization methods. To circumvent these difficulties, a surrogate-model-based derivative-free optimization algorithm is proposed for solving the DOO problem. In order to establish the surrogate model with the least number of function evaluations, a subset selection strategy is designed to find a sparse structure for the optimization problem, based on which a set of simple bases is determined to establish the surrogate model. Moreover, this also reduces the scale of the parameter optimization problem. Numerical experiments on actual production data verify the applicability and effectiveness of the proposed method.

An Application of Surrogate and Resampling for the Optimization of Success Probability from Binary-Response Type Simulation

Since traditional derivative-based optimization for noisy simulation shows bad performance, evolutionary algorithms are considered as substitutes. Especially in case when outputs are binary, more simulation trials are needed to get near-optimal solution since the outputs are discrete and have high and heterogeneous variance. In this paper, we propose a genetic algorithm called SARAGA which adopts dynamic resampling and fitness approximation using surrogate. SARAGA reduces unnecessary numbers of expensive simulations to estimate success probabilities estimated from binary simulation outputs. SARAGA allocates number of samples to each solution dynamically and sometimes approximates the fitness without additional expensive experiments. Experimental results show that this novel approach is effective and proper hyper parameter choice of surrogate and resampling can improve the performance of algorithm.

Defect reconstruction in a two-dimensional semi-analytical waveguide model via derivative-based optimization.

This paper considers an indirect measurement approach to reconstruct a defect in a two-dimensional waveguide model for a non-destructive ultrasonic inspection via derivative-based optimization. The propagation of the mechanical waves is simulated by the scaled boundary finite element method that builds on a semi-analytical approach. The simulated data are then fitted to given data associated with the reflected waves from a defect which is to be reconstructed. For this purpose, we apply an iteratively regularized Gauss-Newton method in combination with algorithmic differentiation to provide the required derivative information accurately and efficiently. We present numerical results for three kinds of defects, namely, a crack, delamination, and corrosion. The objective function and the properties of the reconstruction method are investigated. The examples show that the parameterization of the defect can be reconstructed efficiently as well as robustly in the presence of noise.

Using first-order information in direct multisearch for multiobjective optimization

Derivatives are an important tool for single-objective optimization. In fact, it is commonly accepted that derivative-based methods present a better performance than derivative-free optimization approaches. In this work, we will show that the same does not always apply to multiobjective derivative-based optimization, when the goal is to compute an approximation to the complete Pareto front of a given problem. The competitiveness of direct multisearch (DMS), a robust and efficient derivative-free optimization algorithm, will be stated for derivative-based multiobjective optimization (MOO) problems, by comparison with MOSQP, a state-of-art derivative-based MOO solver. We will then assess the potential enrichment of adding first-order information to the DMS framework. Derivatives will be used to prune the positive spanning sets considered at the poll step of the algorithm. The role of ascent directions, that conform to the geometry of the nearby feasible region, will then be highlighted.

A Fast Method for Fuzzy Rules Learning with Derivative-Free Optimization by Formulating Independent Evaluations of Each Fuzzy Rule

A method is proposed for evaluating fuzzy rules independently of each other in fuzzy rules learning. The proposed method is named α-FUZZI-ES (α-weight-based fuzzy-rule independent evaluations) in this paper. In α-FUZZI-ES, the evaluation value of a fuzzy system is divided out among the fuzzy rules by using the compatibility degrees of the learning data. By the effective use of α-FUZZI-ES, a method for fast fuzzy rules learning is proposed. This is named α-FUZZI-ES learning (α-FUZZI-ES-based fuzzy rules learning) in this paper. α-FUZZI-ES learning is especially effective when evaluation functions are not differentiable and derivative-based optimization methods cannot be applied to fuzzy rules learning. α-FUZZI-ES learning makes it possible to optimize fuzzy rules independently of each other. This property reduces the dimensionality of the search space in finding the optimum fuzzy rules. Thereby, α-FUZZI-ES learning can attain fast convergence in fuzzy rules optimization. Moreover, α-FUZZI-ES learning can be efficiently performed with hardware in parallel to optimize fuzzy rules independently of each other. Numerical results show that α-FUZZI-ES learning is superior to the exemplary conventional scheme in terms of accuracy and convergence speed when the evaluation function is non-differentiable.

Figures of merit for stellarators near the magnetic axis

A new paradigm for rapid stellarator configuration design has been recently demonstrated, in which the shapes of quasisymmetric or omnigenous flux surfaces are computed directly using an expansion in small distance from the magnetic axis. To further develop this approach, here we derive several other quantities of interest that can be rapidly computed from this near-axis expansion. First, the $\boldsymbol {\nabla }\boldsymbol {B}$ and $\boldsymbol {\nabla }\boldsymbol {\nabla }\boldsymbol {B}$ tensors are computed, which can be used for direct derivative-based optimization of electromagnetic coil shapes to achieve the desired magnetic configuration. Moreover, if the norm of these tensors is large compared with the field strength for a given magnetic field, the field must have a short length scale, suggesting it may be hard to produce with coils that are suitably far away. Second, we evaluate the minor radius at which the flux surface shapes would become singular, providing a lower bound on the achievable aspect ratio. This bound is also shown to be related to an equilibrium beta limit. Finally, for configurations that are constructed to achieve a desired magnetic field strength to first order in the expansion, we compute the error field that arises due to second-order terms.

Iterative ensemble Kalman methods: A unified perspective with some new variants

<p style='text-indent:20px;'>This paper provides a unified perspective of iterative ensemble Kalman methods, a family of derivative-free algorithms for parameter reconstruction and other related tasks. We identify, compare and develop three subfamilies of ensemble methods that differ in the objective they seek to minimize and the derivative-based optimization scheme they approximate through the ensemble. Our work emphasizes two principles for the derivation and analysis of iterative ensemble Kalman methods: statistical linearization and continuum limits. Following these guiding principles, we introduce new iterative ensemble Kalman methods that show promising numerical performance in Bayesian inverse problems, data assimilation and machine learning tasks.</p>

A Bilevel Learning Model and Algorithm for Self-Organizing Feed-Forward Neural Networks for Pattern Classification.

Conventional artificial neural network (ANN) learning algorithms for classification tasks, either derivative-based optimization algorithms or derivative-free optimization algorithms work by training ANN first (or training and validating ANN) and then testing ANN, which are a two-stage and one-pass learning mechanism. Thus, this learning mechanism may not guarantee the generalization ability of a trained ANN. In this article, a novel bilevel learning model is constructed for self-organizing feed-forward neural network (FFNN), in which the training and testing processes are integrated into a unified framework. In this bilevel model, the upper level optimization problem is built for testing error on testing data set and network architecture based on network complexity, whereas the lower level optimization problem is constructed for network weights based on training error on training data set. For the bilevel framework, an interactive learning algorithm is proposed to optimize the architecture and weights of an FFNN with consideration of both training error and testing error. In this interactive learning algorithm, a hybrid binary particle swarm optimization (BPSO) taken as an upper level optimizer is used to self-organize network architecture, whereas the Levenberg-Marquardt (LM) algorithm as a lower level optimizer is utilized to optimize the connection weights of an FFNN. The bilevel learning model and algorithm have been tested on 20 benchmark classification problems. Experimental results demonstrate that the bilevel learning algorithm can significantly produce more compact FFNNs with more excellent generalization ability when compared with conventional learning algorithms.

Near zero-bias MIIM diode based on TiO2/ZnO for energy harvesting applications

Energy harvesting rectennas require ultrafast rectifying diodes that are efficiently matched to the optical nanoantenna. These diodes should possess low on-resistance and high responsivity. Here, we introduce a metal-insulator-metal diode composed of a new material, Ti-TiO2-Al. This diode has a 1.0 nm ultrathin insulator layer fabricated using atomic layer deposition (ALD). It has a zero-bias resistance of 275 Ω and a maximum responsivity of 3.1 A/W. To further improve its performance, another ultrathin layer of ZnO was added. The proposed Ti-TiO2/ZnO-Al metal-insulator-insulator-metal diode has a zero-bias resistance of 312 Ω and a maximum responsivity of 5.1 A/W. The two types of diodes are fabricated on a SiO2 substrate using conventional photolithography and ALD. Between 20 °C and 55 °C, the I-V characteristics did not show much temperature dependence. The effective barrier height, dielectric constant, and electron effective mass in each insulator are extracted using a constrained and derivative-based optimization algorithm.

AdequacyModel: An R package for probability distributions and general purpose optimization.

Several lifetime distributions have played an important role to fit survival data. However, for some of these models, the computation of maximum likelihood estimators is quite difficult due to presence of flat regions in the search space, among other factors. Several well-known derivative-based optimization tools are unsuitable for obtaining such estimates. To circumvent this problem, we introduce the AdequacyModel computational library version 2.0.0 for the R statistical environment with two major contributions: a general optimization technique based on the Particle Swarm Optimization (PSO) method (with a minor modification of the original algorithm) and a set of statistical measures for assessment of the adequacy of the fitted model. This library is very useful for researchers in probability and statistics and has been cited in various papers in these areas. It serves as the basis for the Newdistns library (version 2.1) published in an impact journal in the area of computational statistics, see https://CRAN.R-project.org/package=Newdistns. It is also the basis of the Wrapped library (version 2.0), see https://CRAN.R-project.org/package=Wrapped. A third package making use of the AdequacyModel library can be found in https://CRAN.R-project.org/package=sglg. In addition, the proposed library has proved to be very useful for maximizing log-likelihood functions with complex search regions. The library provides a greater control of the optimization process by introducing a stop criterion based on a minimum number of iterations and the variance of a given proportion of optimal values. We emphasize that the new library can be used not only in statistics but in physics and mathematics as proved in several examples throughout the paper.

Indirect inference with a non-smooth criterion function

Indirect inference requires simulating realizations of endogenous variables from the model under study. When the endogenous variables are discontinuous functions of the model parameters, the resulting indirect inference criterion function is discontinuous and does not permit the use of derivative-based optimization routines. Using a change of variables technique, we propose a novel simulation algorithm that alleviates the discontinuities inherent in such indirect inference criterion functions, and permits the application of derivative-based optimization routines to estimate the unknown model parameters. Unlike competing approaches, this approach does not rely on kernel smoothing or bandwidth parameters. Several Monte Carlo examples that have featured in the literature on indirect inference with discontinuous outcomes illustrate the approach, and demonstrate the superior performance of this approach over existing alternatives.

Generalized Linear Quaternion Complementary Filter for Attitude Estimation From Multisensor Observations: An Optimization Approach

Focusing on generalized sensor combinations, this paper deals with the attitude estimation problem using a linear complementary filter (CF). The quaternion observation model is obtained via a gradient descent algorithm. An additive measurement model is then established according to derived results. The filter is named as the generalized CF where the observation model is simplified as a linear one that is quite different from previous-reported brute-force nonlinear results. Moreover, we prove that representative derivative-based optimization algorithms are essentially equivalent to each other. Derivations are given to establish the state model based on the quaternion kinematic equation. The proposed algorithm is validated under several experimental conditions involving the free-living environment, harsh external field disturbances, and aerial flight test aided by robotic vision. Using the specially designed experimental devices, data acquisition and algorithm computations are performed to give comparisons on accuracy, robustness, time-consumption, and so on with representative methods. The results show that not only the proposed filter can give fast, accurate, and stable estimates in terms of various sensor combinations but also produces robust attitude estimation in the scenario of harsh situations, e.g., irregular magnetic distortion. Note to Practitioners —Multisensor attitude estimation is a crucial technique in robotic devices. Many existing methods focus on the orientation fusion of specific sensor combinations. In this paper, we make the problem more concise. The results given in this paper are very general and can significantly decrease the space consumption and computation burden without losing the original estimation accuracy. Such performance will be of benefit to robotic platforms requiring flexible and easy-to-tune attitude estimation in the future.

Model implementation and verification of the envelope, HVAC and controller of an office building in Modelica

Modelica is a promising open-source modelling language for simulation and derivative-based optimization of multi-zone buildings, including detailed pressure-driven flow networks and building controllers. Such Modelica models have however not yet been demonstrated for buildings with many zones and complex HVAC systems. This paper presents and provides implementation details of the simulation model of a 32-zones building model in Modelica, with pressure-driven flow networks and a building controller, demonstrating the potential of Modelica. The modelling approach and computational aspects are presented. Furthermore, the model is calibrated and verified using on-site measurement data, which shows that, for the presented case study, HVAC can be modelled with errors up to only a few percent, while larger errors are observed for the building envelope model. A detailed building description is provided as supplementary material.

Evaluation of Derivative-Free Optimizers for Parameter Estimation in Systems Biology

Derivative-free optimization can be used to estimate parameters without computing derivatives. As there exist many methods, it is unclear which to use in practice. Here, we present two comparative studies of 18 state-of-the-art methods: Firstly, we evaluate them on a set of 466 classic optimization test problems of dimension 2 to 300. Secondly, we study their performance in parameter estimation on 8 ODE models of biological processes, comparing them to state-of-the-art derivative-based optimization. We observe that different problem features necessitate the use of different methods, for which we can give suggestions based on our findings. Our analysis suggests that classic test problems are not representative for problems in systems biology. For ODE models, we find that purely derivative-free methods are for most problems not reliable or at least inferior to derivative-based methods.

On Modelling of Data-Driven Monotone Zero-Order TSK Fuzzy Inference Systems using a System Identification Framework

A system identification-based framework is used to develop monotone fuzzy If-Then rules for formulating monotone zero-order Takagi-Sugeno-Kang (TSK) fuzzy inference systems (FISs) in this paper. Convex and normal trapezoidal and triangular fuzzy sets, together with a strong fuzzy partition strategy (either fixed or adaptive), is adopted. By coupling the strong fuzzy partition with a set of complete and monotone fuzzy If-Then rules, a monotone TSK FIS model can be guaranteed. We show that when a clean multiattribute monotone dataset is used, a system identification-based framework does not guarantee the production of monotone fuzzy If-Then rules, which leads to nonmonotone TSK FIS models. This is a new learning phenomenon that needs to be scrutinized when we design data-based monotone TSK FIS models. Two solutions are proposed: 1) a new monotone fuzzy rule relabeling-based method and 2) a constrained derivative-based optimization method. A new modeling framework with an adaptive fuzzy partition is evaluated. The results indicate that TSK FIS models with better accuracy (a lower sum square error) and a good degree of monotonicity (measured with a monotonicity test) are achieved. In short, the main contributions of this study are validation of the new learning phenomenon and introduction of useful methods for developing data-based monotone TSK FIS models.

A hybrid MILP and IPM approach for dynamic economic dispatch with valve-point effects

Dynamic economic dispatch with valve-point effects (DED-VPE) is a non-convex and non-differentiable optimization problem that is difficult to solve efficiently. In this paper, a hybrid approach combining mixed-integer linear programming (MILP) and the interior point method (IPM), abbreviated as MILP-IPM, is proposed to solve a DED-VPE problem in which complicated transmission loss is also included. Due to the non-differentiable nature of the DED-VPE problem, the classical derivative-based optimization methods can no longer be used. With the help of model reformulation, a differentiable non-linear programming (NLP) formulation that can be directly solved using the IPM is derived. However, if the DED-VPE problem is solved using the IPM in a single step, the optimization can easily become trapped in a poor local optimum due to its non-convex nature and multiple local minima. To obtain a better solution, an MILP method is required to solve the DED-VPE problem without transmission loss, yielding a good initial point for the IPM to improve the quality of the solution. Simulation results demonstrate the validity and effectiveness of the proposed MILP-IPM approach for solving the DED-VPE problem.

Global Optimization of Reactive Distillation Processes Using Bat Algorithm

Reactive distillation (RD) is an important process intensification approach with several advantages. It can improve the reaction selectivity and yield, overcome the thermodynamic restrictions, and reduce the cost/energy. However, the optimal design of RD relies on highly nonlinear and multivariable optimization including continuous and integer design variables. The objective function is generally non-convex with several constraints. For this problem, the conventional derivative-based optimization algorithms are faced with convergence problems and fail to guarantee the global optimal solution. Stochastic optimization algorithms appear to be a better alternative for the optimal design of RD because of the high robustness and efficiency. Bat algorithm (BA), which combines advantages of other existing algorithms, is a potential stochastic optimization algorithm. In this work, the BA was used to optimize the RD for the production of methyl acetate (MeAc). The link between Matlab and Aspen Plus was also created to ensure that each solution was provided from rigorous simulations. The total annual cost (TAC) was set as the objective function. Product purity constraints were achieved through Aspen plus instead of algorithms to simplify the process. BA can find the global optimal solution within less computation time than other stochastic algorithms or sequential optimization.