NLP Solver Research Articles

Multistart algorithms for seeking feasibility

This paper describes modifications to two multistart algorithms for global optimization which enable them to find feasible solutions to a system of nonlinear constraints more efficiently. The multistart algorithms, called OptQuest-NLP (OQNLP) and Multistart-NLP (MSNLP), start a local NLP Solver from a set of starting points and return the best solution found. Candidate starting points are generated either by a scatter search heuristic or by a randomized process. Two adaptive filters choose a small subset of the candidate points as starting points. The modifications to facilitate feasibility seeking include replacing the exact penalty function used to measure the goodness of a starting point with the sum of infeasibilities, and terminating when a feasible solution is found. We describe experimental results on a large and diverse set of smooth nonlinear nonconvex problems coded in the GAMS modeling language. These are chosen so that a single application of a selected solver from the user-specified starting point terminates infeasible, yet the problems all have feasible solutions. Our results show that MSNLP's feasibility mode is able to find feasible solutions to almost all problems. It is moderately faster than MSNLP not using feasibility mode, and is somewhat better a finding feasible solutions when they exist. It is now an option within MSNLP, and can be invoked by inserting an appropriate record into the options file.

Improved Filters and Randomized Drivers for Multi-start Global Optimization

We describe and test 2 types of improvements to the multistart heuristic framework for smooth global optimization presented in [8]. This starts a gradient based local NLP solver from a filtered subset of points generated by a starting point generator or driver, and has been available as both a standalone and GAMS solver since 2003. Changes to the distance and merit filters make them theoretically sounder and more adaptive to the problem instance. Stochastic drivers are proposed whose performance is comparable to the OptQuest scatter search implementation, if reasonably tight bounds on all variables are imposed. Tests on two large and diverse sets of test problems (56 problems with over 100 variables, and 18 with over 1000) show that best known solutions to almost all problems can be obtained with a median of about 30 solver calls. The best solution is located in the first ten calls in about 93% of the problems, making the procedure attractive within a branch and bound framework. Further, allowing more time and iterations usually yields improved solutions to problems not solved with default limits.

Path relinking and GRG for artificial neural networks

Artificial neural networks (ANN) have been widely used for both classification and prediction. This paper is focused on the prediction problem in which an unknown function is approximated. ANNs can be viewed as models of real systems, built by tuning parameters known as weights. In training the net, the problem is to find the weights that optimize its performance (i.e., to minimize the error over the training set). Although the most popular method for training these networks is back propagation, other optimization methods such as tabu search or scatter search have been successfully applied to solve this problem. In this paper we propose a path relinking implementation to solve the neural network training problem. Our method uses GRG, a gradient-based local NLP solver, as an improvement phase, while previous approaches used simpler local optimizers. The experimentation shows that the proposed procedure can compete with the best-known algorithms in terms of solution quality, consuming a reasonable computational effort.

Scatter Search and Local Nlp Solvers: A Multistart Framework for Global Optimization

The algorithm described here, called OptQuest/NLP or OQNLP, is a heuristic designed to find global optima for pure and mixed integer nonlinear problems with many constraints and variables, where all problem functions are differentiable with respect to the continuous variables. It uses OptQuest, a commercial implementation of Scatter Search developed by OptTek Systems, Inc., to provide starting points for any gradient-based local NLP solver. This solver seeks a local solution from a subset of these points, holding discrete variables fixed. The procedure is motivated by our desire to combine the superior accuracy and feasibility-seeking behavior of gradient-based local NLP solvers with the global optimization abilities of OptQuest. Computational results include 155 smooth NLP and MINLP problems due to Floudas et al., most with both linear and nonlinear constraints, coded in the GAMS modeling language. Some are quite large for global optimization, with over 100 variables and 100 constraints. Global solutions to almost all problems are found in a small number of local solver calls, often one or two.

Gradient Estimation Schemes for Noisy Functions

In this paper, we analyze different schemes for obtaining gradient estimates when the underlying functions are noisy. Good gradient estimation is important e.g. for nonlinear programming solvers. As error criterion, we take the norm of the difference between the real and estimated gradients. The total error can be split into a deterministic error and a stochastic error. For three finite-difference schemes and two design of experiments (DoE) schemes, we analyze both the deterministic errors and stochastic errors. We derive also optimal stepsizes for each scheme, such that the total error is minimized. Some of the schemes have the nice property that this stepsize minimizes also the variance of the error. Based on these results, we show that, to obtain good gradient estimates for noisy functions, it is worthwhile to use DoE schemes. We recommend to implement such schemes in NLP solvers.

Approximating Unknown Mappings: An Experimental Evaluation

Different methodologies have been introduced in recent years with the aim of approximating unknown functions. Basically, these methodologies are general frameworks for representing non-linear mappings from several input variables to several output variables. Research into this problem occurs in applied mathematics (multivariate function approximation), statistics (nonparametric multiple regression) and computer science (neural networks). However, since these methodologies have been proposed in different fields, most of the previous papers treat them in isolation, ignoring contributions in the other areas. In this paper we consider five well known approaches for function approximation. Specifically we target polynomial approximation, general additive models (Gam), local regression (Loess), multivariate additive regression splines (Mars) and artificial neural networks (Ann). Neural networks can be viewed as models of real systems, built by tuning parameters known as weights. In training the net, the problem is to find the weights that optimize its performance (i.e. to minimize the error over the training set). Although the most popular method for Ann training is back propagation, other optimization methods based on metaheuristics have recently been adapted to this problem, outperforming classical approaches. In this paper we propose a short term memory tabu search method, coupled with path relinking and BFGS (a gradient-based local NLP solver) to provide high quality solutions to this problem. The experimentation with 15 functions previously reported shows that a feed-forward neural network with one hidden layer, trained with our procedure, can compete with the best-known approximating methods. The experimental results also show the effectiveness of a new mechanism to avoid overfitting in neural network training.

Dynamic Optimization of HIPS Open-Loop Unstable Polymerization Reactors

We consider dynamic optimization strategies for grade transitions for high-impact polystyrene reactors. Because our desired operating conditions are at unstable points, we apply a simultaneous dynamic optimization (SDO) approach, where state and control variables in the optimal control problem are discretized and a large-scale nonlinear programming solver is applied. For this purpose, we consider Radau collocation on finite elements and the IPOPT NLP solver. In addition, we describe the stability of the SDO strategy through the presentation of dichotomy properties for boundary-value problems. The resulting SDO approach is then demonstrated on a wide variety of operating scenarios for the high-impact polystyrene (HIPS) reactor, with highly reliable and efficient performance results.

Reformulation descent applied to circle packing problems

Several years ago classical Euclidean geometry problems of densest packing of circles in the plane have been formulated as nonconvex optimization problems, allowing to find heuristic solutions by using any available NLP solver. In this paper we try to improve this procedure. The faster NLP solvers use first order information only, so stop in a stationary point. A simple switch from Cartesian coordinates to polar or vice versa, may destroy this stationarity and allow the solver to descend further. Such formulation switches may of course be iterated. For densest packing of equal circles into a unit circle, this simple feature turns out to yield results close to the best known, while beating second order methods by a time-factor well over 100. This technique is formalized as a general reformulation descent (RD) heuristic, which iterates among several formulations of the same problem until local searches obtain no further improvement. We also briefly discuss how RD might be used within other metaheuristic schemes.

Canonical coordinates method for equality-constrained nonlinear optimization

Feasible-points methods have several appealing advantages over the infeasible-points methods for solving equality-constrained nonlinear optimization problems. The known feasible-points methods however solve, often large, systems of nonlinear constraint equations in each step in order to maintain feasibility. Solving nonlinear equations in each step not only slows down the algorithms considerably, but also the large amount of floating-point computation involved introduces considerable numerical inaccuracy into the overall computation. As a result, the commercial software packages for equality-constrained optimization are slow and not numerically robust. We present a radically new approach to maintaining feasibility––called the canonical coordinates method (CCM). The CCM, unlike previous methods, does not adhere to the coordinate system used in the problem specification. Rather, as the algorithm progresses CCM dynamically chooses, in each step, a coordinate system that is most appropriate for describing the local geometry around the current iterate. By dynamically changing the coordinate system to suit the local geometry, the CCM is able to maintain feasibility in equality-constrained nonlinear optimization without having to solve systems of nonlinear equations. We describe the CCM and present a proof of its convergence. We also present a few numerical examples which show that CCM can solve, in very few iterations, problems that cannot be solved using the commercial NLP solver in MATLAB 6.1.

Optimization of a heat exchanger network superstructure using nonlinear programming

A heat exchanger network design for a particular set of hot and cold stream duties requires multiple stream splits for minimum energy use. A modern NLP solver, FilterSQP, is applied to minimize a total cost function that takes account of capital and energy costs. The best solution contains fewer exchangers than the initial network. There are multiple local optima at various cost levels. They contain different subsets of exchangers of the initial design, which constitutes a partial superstructure for the problem. Several optimization alternatives are examined: for the model formulation, leading to problems with different optimum costs, for the way the objective is written, which affects the formal degree of nonlinearity in the equations or objective, and for the way the solver operates. For each combination of options, tests were run using widely different initial values for the Trust Region size, a parameter in the solver. FilterSQP solved the network with impressive reliability and efficiency from several starting guesses, some of which were highly arbitrary. From even the most unpromising initial guesses, the solver converged to a local optimum in nearly all cases.

Decomposition algorithms for on-line estimation with nonlinear DAE models

In a previous paper we presented an algorithm for solving the large nonlinear problems that arise in dynamic state and parameter estimation. These problems are described by a set of Ordinary Differential Equations (ODE) subject to initial values, discretized and solved under a simultaneous solution strategy within an Error-in-all-Variable-Measurements (EVM) framework. In this paper, we extend this work to systems described by Differential and Algebraic Equations (DAE), for both initial value (IVP) and boundary value (BVP) problems. The resulting discretized NLP problems are usually large both in the number of variables and the number of degrees-of-freedom, and the size of the problem grows linearly with the number of data samples or the number of discretizations, making them difficult to solve with general purpose NLP solvers. We discretize the DAE model with an implicit Runge-Kutta scheme (IRK) so that higher index and stiff problems can be solved, and use the SQP method to solve the resulting NLP. Under the assumption that the measurement errors are independently distributed, the optimality conditions for each data set at the QP subproblem level are decoupled so that the first order conditions in the state variables, input variables and the stage derivatives can be solved recursively and expressed as functions of the optimality conditions in the parameters, thus reducing the size of the QP subproblem and turning the solution effort linear with both the number of sample points and the number of discretization points. We use two decoupling procedures depending on whether the DAE system is expressed as an IVP or a BVP. For the IVP we extend the technique used in our previous paper and use an affine transform which expresses the directions in the variables and the multipliers as functions of the steps in the parameters. For the BVP we use a selected variable elimination procedure so that the optimality conditions in the state and input variables, stage derivatives and multipliers of the QP subproblem will be transformed into an almost block diagonal system of linear equations. We then solve this system using SOLVEBLOCK, a solution method linear with the number of blocks to reduce the size of the QP subproblem to the dimension of the parameters. As seen in our examples, these approaches are therefore much faster than general purpose NLP solvers. In this paper we also present a case where an IVP is unstable and difficult to solve, but stable and well behaved when rewritten as a BVP.

Dynamic simulation and optimization with inequality path constraints

A novel and general method for numerical integration of ODEs or DAEs with inequality path constraints involving state variables is proposed. In general, active inequality path constraints involving state variables produce high-index DAEs, which complicate the solution of dynamic simulation and optimization problems. Current DAE integrators can handle only certain limited classes of high-index problems. However, the recently developed method of dummy derivatives can be used to derive an index-1 problem with the same solution set as the high-index problem. This permits the use of standard DAE codes for integrating high-index DAEs, although the equivalent index-1 DAE has a larger number of equations than the original DAE system. Our method detects the activation and deactivation of inequality path constraints during integration, and solves the resulting high-index DAE system as necessary. In addition to allowing the solution of dynamic simulation problems with inequality path constraints, we show that this method simplifies the solution of dynamic optimization problems using the control parameterization method. The method allows us to handle the inequality path constraints involving state variables within the DAE integrator, resulting in fewer objective function and gradient evaluations for the NLP solver, and reducing the time necessary to solve the dynamic optimization problem.

Tailoring optimization algorithms to process applications

Over the past decade the application of efficient nonlinear programming tools has become a powerful and established strategy for process analysis and design. While simple coupling of standard optimization codes to process applications has demonstrated the effectiveness of this approach, further development is needed for the efficient large-scale use of these tools. This is particularly true for dynamic optimization problems and on-line applications, as well as design problems with complex models. This paper centers on exploiting the structure of process optimization problems. Here we consider the tailoring of Newton-type optimization algorithms for various process applications. In particular, the Successive Quadratic Programming (SQP) algorithm has been successful over the past decade; it has also been extended in a number of ways to optimization problems involving several thousand variables. After describing a large-scale, general purpose NLP solver, we briefly describe specialized SQP algorithms for the following problem classes: optimization of complex, structured process models with few degrees of freedom optimization of dynamic systems, where structured sets of nonlinear equations are created from discretization of differential-algebraic systems (DAEs) treatment of parameter estimation problems with many degrees of freedom Each of these aspects is illustrated with a small process application and efficient techniques for handling these problems are outlined. Also, these methods are briefly compared to general purpose algorithms in order to demonstrate the effectiveness of this approach.