Trust-region Step Research Articles

Efficient Multi-Stage Conjugate Gradient for Trust Region Step

The trust region step problem, by solving a sphere constrained quadratic programming, plays a critical role in the trust region Newton method. In this paper, we propose an efficient Multi-Stage Conjugate Gradient (MSCG) algorithm to compute the trust region step in a multi-stage manner. Specifically, when the iterative solution is in the interior of the sphere, we perform the conjugate gradient procedure. Otherwise, we perform a gradient descent procedure which points to the inner of the sphere and can make the next iterative solution be a interior point. Subsequently, we proceed with the conjugate gradient procedure again. We repeat the above procedures until convergence. We also present a theoretical analysis which shows that the MSCG algorithm converges. Moreover, the proposed MSCG algorithm can generate a solution in any prescribed precision controlled by a tolerance parameter which is the only parameter we need. Experimental results on large-scale text data sets demonstrate our proposed MSCG algorithm has a faster convergence speed compared with the state-of-the-art algorithms.

Optimization of pressure swing adsorption via a trust-region filter algorithm and equilibrium theory

Optimization of pressure swing adsorption (PSA) remains a challenging task, as these are periodic dynamic systems governed by nonlinear PDE systems. This study develops an optimization strategy that incorporates reduced PSA models and also samples information from a high-fidelity ’truth’ model. The reduced model is based on equilibrium theory and can be applied to optimize a large variety of cyclic adsorption processes. The optimization is performed via a trust-region filter (TRF) method. The TRF method uses reduced models to minimize the information required from the high-fidelity adsorption model during mathematical optimization. Moreover, this method guarantees convergence to the optimum of the high-fidelity model. This approach is applied to optimize a 4 column pressure swing adsorption process, modeled by partial differential algebraic equations, for the separation of a CO2/CH4 mixture. The results show that the reduced model significantly reduces the computational time of the method’s trust-region step compared to previous studies.

A trust region method with project step for bound constrained optimization without compact condition

ABSTRACT In this paper, we propose a trust region method for the solution of the nonlinear optimization problem with bound constraints. The method differs from the existing trust region method in that we use the trust region step as a trial step and the project step as a correction step. We prove that the generated sequence is bounded automatically without the compact assumption and obtain global convergence of the proposed algorithm. Numerical results are given to show the efficiency when combined with suitable quasi-Newton update such as BFGS and Barzilai-Borwein update for the Hessian matrix.

An efficient two-step trust-region algorithm for exactly determined consistent systems of nonlinear equations

In this paper, we propose a modified two-step trust-region algorithm for solving nonlinear systems. Two-step trust-region algorithms, at every iteration, use a trust-region step and an approximate step by saving the Jacobian matrix to have fewer computations. We introduce a convex combination to modify the trust-region subproblems along with an additional criterion to verify whether the first step is accepted or not. We establish global and quadratic convergence of the algorithm under some mild assumptions. Numerical results show that the modified algorithm is efficient and promising.

A Radial Boundary Intersection aided interior point method for multi-objective optimization

We propose a novel multi-objective optimization technique combining non-convex Radial Boundary Intersection based decomposition with an Interior Point method (which utilizes both line search and trust region steps) suitable for non-convex nonlinear optimization. Radial Boundary Intersection decomposes the multi-objective optimization problem into subproblems which are concerned with finding the solutions closest to a reference point along equally spaced lines emanating radially outwards from the latter point. The proposed approach is found to be able to generate good approximations of the Pareto front (including the periphery) by generating a sufficiently diverse set of Pareto optimal solutions. The proposed method is extensively tested on a large number of recent benchmark problems and real world problems and the performance is found to be favorable in comparison to those of some of the cutting-edge stochastic/evolutionary optimization algorithms that are commonly used to solve non-convex multi-objective optimization problems.

TRUST-REGION-BASED METHODS FOR NONLINEAR PROGRAMMING: RECENT ADVANCES AND PERSPECTIVES

The aim of this text is to highlight recent advances of trust-region-based methods for nonlinear programming and to put them into perspective. An algorithmic framework provides a ground with the main ideas of these methods and the related notation. Specific approaches concerned with handling the trust-region subproblem are recalled, particularly for the large scale setting. Recent contributions encompassing the trust-region globalization technique for nonlinear programming are reviewed, including nonmonotone acceptance criteria for unconstrained minimization; the adaptive adjustment of the trust-region radius; the merging of the trust-region step into a line search scheme, and the usage of the trust-region elements within derivative-free optimization algorithms.

Globally convergent DC trust-region methods

In this paper, we investigate the use of DC (Difference of Convex functions) models and algorithms in the application of trust-region methods to the solution of a class of nonlinear optimization problems where the constrained set is closed and convex (and, from a practical point of view, where projecting onto the feasible region is computationally affordable). We consider DC local models for the quadratic model of the objective function used to compute the trust-region step, and apply a primal-dual subgradient method to the solution of the corresponding trust-region subproblems. One is able to prove that the resulting scheme is globally convergent to first-order stationary points. The theory requires the use of exact second-order derivatives but, in turn, the computation of the trust-region step asks only for one projection onto the feasible region (in comparison to the calculation of the generalized Cauchy point which may require more). The numerical efficiency and robustness of the proposed new scheme when applied to bound-constrained problems is measured by comparing its performance against some of the current state-of-the-art nonlinear programming solvers on a vast collection of test problems.

Robust and efficient Trust-Region based stability analysis and multiphase flash calculations

Multiphase flash calculations and phase stability analysis are central in compositional reservoir and chemical process simulators. For instance, in some simulations, a huge amount of phase equilibrium calculations is required (the most important part of the computational effort). Moreover, a single failure may cause significant error propagations leading to false solutions. Thus, it is imperative that calculation algorithms are efficient and highly robust. The most difficult regions in mixture phase envelopes are in the vicinity of singularities: critical points for flash calculations and the stability test limit locus for stability analysis. For these conditions, all algorithms have difficulties to converge. Traditionally, a number of successive substitution iterations (SSI) are performed before switching to the second-order Newton method (many SSI iteration may be required before switch very close to singularities). The Trust-Region method has the advantage of performing a Newton step whenever the Hessian is definite positive; otherwise, the Trust-Region corrects the Hessian matrix by adding a diagonal element to make it positive definite, thus a descent direction is guaranteed. The Trust-Region limits the solution within a trust-radius, which is updated automatically at each iteration level, depending on the quality of the quadratic approximation. If the function is convex, the trust-radius enables larger changes in iteration variables, otherwise restricted steps are used to ensure a progress towards the solution. The proposed Trust-Region algorithm, as well as a hybrid methodology that combines SSI, Newton and Trust-Region steps, are tested for multiphase flash calculations and stability analysis on a variety of mixtures involving hydrocarbon components, carbon dioxide and hydrogen sulfide, exhibiting complicated phase envelopes. The proposed method compares favorably to the widely used SSI-Newton methods with various independent variables. The more difficult a test point is, the more spectacular the algorithm acts from both efficiency and reliability perspectives.

An adaptive nonmonotone trust-region method with curvilinear search for minimax problem

In this paper we propose an adaptive nonmonotone algorithm for minimax problem. Unlike traditional nonmonotone method, the nonmonotone technique applied to our method is based on the nonmonotone technique proposed by Zhang and Hager [H.C. Zhang, W.W. Hager, A nonmonotone line search technique and its application to unconstrained optimization, SIAM J. Optim. 14(4)(2004) 1043–1056] instead of that presented by Grippo et al. [L. Grippo, F. Lampariello, S. Lucidi, A nonmonotone line search technique for Newton’s method, SIAM J. Numer. Anal. 23(4)(1986) 707–716]. Meanwhile, by using adaptive technique, it can adaptively perform the nonmonotone trust-region step or nonmonotone curvilinear search step when the solution of subproblems is unacceptable. Global and superlinear convergences of the method are obtained under suitable conditions. Preliminary numerical results are reported to show the effectiveness of the proposed algorithm.

Trust region-based optimization of PKI neural models for RF/microwave devices

This article proposes a new trust region-based optimization technique for Radio Frequency (RF)/microwave devices. The proposed approach is apt for modeling scenarios, where standard ANN multilayer perceptron (MLP) and Prior Knowledge Input (PKI) models fail to deliver a satisfactory model. This approach feeds output of standard ANN model as knowledge input to PKI model. The ANN model and the PKI model form a symbiotic pair to yield accurate results. In this paper, the dogleg routine is exploited in the process of optimization to obtain valid trust region steps. The proposed method is compared with sensitivity technique via several RF/microwave components. © 2012 Wiley Periodicals, Inc. Int J RF and Microwave CAE, 2013.

A surrogate management framework using rigorous trust-region steps

Surrogate models are frequently used in the optimization engineering community as convenient approaches to deal with functions for which evaluations are expensive or noisy, or lack convexity. These methodologies do not typically guarantee any type of convergence under reasonable assumptions. In this article, we will show how to incorporate the use of surrogate models, heuristics, or any other process of attempting a function value decrease in trust-region algorithms for unconstrained derivative-free optimization, in a way that global convergence of the latter algorithms to stationary points is retained. Our approach follows the lines of search/poll direct-search methods and corresponding surrogate management frameworks, both in algorithmic design and in the form of organizing the convergence theory.

On the convergence of trust region algorithms for unconstrained minimization without derivatives

We consider iterative trust region algorithms for the unconstrained minimization of an objective function $F ( \underline{x})$ , $\underline{x}\in \mathcal{R}^{n}$ , when F is differentiable but no derivatives are available, and when each model of F is a linear or a quadratic polynomial. The models interpolate F at n+1 points, which defines them uniquely when they are linear polynomials. In the quadratic case, second derivatives of the models are derived from information from previous iterations, but there are so few data that typically only the magnitudes of second derivative estimates are correct. Nevertheless, numerical results show that much faster convergence is achieved when quadratic models are employed instead of linear ones. Just one new value of F is calculated on each iteration. Changes to the variables are either trust region steps or are designed to maintain suitable volumes and diameters of the convex hulls of the interpolation points. It is proved that, if F is bounded below, if ?2 F is also bounded, and if the number of iterations is infinite, then the sequence of gradients $\underline{\nabla}F ( \underline{x}_{\,k} )$ , k=1,2,3,?, converges to zero, where $\underline{x}_{\,k}$ is the centre of the trust region of the k-th iteration.

A Sequential Linear Programming Coordination Algorithm for Analytical Target Cascading

Decomposition-based strategies, such as analytical target cascading (ATC), are often employed in design optimization of complex systems. Achieving convergence and computational efficiency in the coordination strategy that solves the partitioned problem is a key challenge. A new convergent strategy is proposed for ATC that coordinates interactions among subproblems using sequential linearizations. The linearity of subproblems is maintained using infinity norms to measure deviations between targets and responses. A subproblem suspension strategy is used to suspend temporarily inclusion of subproblems that do not need significant redesign, based on trust region and target value step size. An individual subproblem trust region method is introduced for faster convergence. The proposed strategy is intended for use in design optimization problems where sequential linearizations are typically effective, such as problems with extensive monotonicities, a large number of constraints relative to variables, and propagation of probabilities with normal distributions. Experiments with test problems show that, relative to standard ATC coordination, the number of subproblem evaluations is reduced considerably while the solution accuracy depends on the degree of monotonicity and nonlinearity.

A Polynomial Predictor-Corrector Trust-Region Algorithm for Linear Programming

In this paper we present a scaling-invariant, interior-point, predictor-corrector type algorithm for linear programming (LP) whose iteration-complexity is polynomially bounded by the dimension and the logarithm of a certain condition number of the LP constraint matrix. At the predictor stage, the algorithm either takes the step along the standard affine scaling (AS) direction or a new trust-region type direction, whose construction depends on a scaling-invariant bipartition of the variables determined by the AS direction. This contrasts with the layered least squares direction introduced in S. Vavasis and Y. Ye [Math. Program., 74 (1996), pp. 79–120], whose construction depends on multiple-layered partitions of the variables that are not scaling-invariant. Moreover, it is shown that the overall arithmetic complexity of the algorithm (weakly) depends on the right-hand side and the cost of the LP in view of the work involved in the computation of the trust region steps.

Iterative Methods for Finding a Trust-region Step

We consider methods for large-scale unconstrained minimization based on finding an approximate minimizer of a quadratic function subject to a two-norm trust-region inequality constraint. The Steihaug-Toint method uses the conjugate-gradient algorithm to minimize the quadratic over a sequence of expanding subspaces until the iterates either converge to an interior point or cross the constraint boundary. The benefit of this approach is that an approximate solution may be obtained with minimal work and storage. However, the method does not allow the accuracy of a constrained solution to be specified. We propose an extension of the Steihaug-Toint method that allows a solution to be calculated to any prescribed accuracy. If the Steihaug-Toint point lies on the boundary, the constrained problem is solved on a sequence of evolving low-dimensional subspaces. Each subspace includes an accelerator direction obtained from a regularized Newton method applied to the constrained problem. A crucial property of this direction is that it can be computed by applying the conjugate-gradient method to a positive-definite system in both the primal and dual variables of the constrained problem. The method includes a parameter that allows the user to take advantage of the tradeoff between the overall number of function evaluations and matrix-vector products associated with the underlying trust-region method. At one extreme, a low-accuracy solution is obtained that is comparable to the Steihaug-Toint point. At the other extreme, a high-accuracy solution can be specified that minimizes the overall number of function evaluations at the expense of more matrix-vector products.

A hybrid algorithm for nonlinear minimax problems

In this paper, a hybrid algorithm for solving finite minimax problem is presented. In the algorithm, we combine the trust-region methods with the line-search methods and curve-search methods. By means of this hybrid technique, the algorithm, according to the specific situation at each iteration, can adaptively performs the trust-region step, line-search step or curve-search step, so as to avoid possibly solving the trust-region subproblems many times, and make better use of the advantages of different methods. Moreover, we use second-order correction step to circumvent the difficulties of the Maratos effect occurred in the nonsmooth optimization. Under mild conditions, we prove that the new algorithm is of global convergence and locally superlinear convergence. The preliminary experiments show that the new algorithm performs efficiently.

A new method of moving asymptotes for large-scale unconstrained optimization

The method of moving asymptotes is known to work well in the context of structural optimization. A new method of moving asymptotes is proposed for solving large-scale unconstrained optimization problems in this paper. In this method, a descending direction is obtained by solving a convex separable subproblem of moving asymptotes in each iteration. New rules for controlling the asymptotes parameters are designed by using the trust region radius and some approximation properties such that the global convergence of this method is obtained. In addition, a linear search technique is inserted in case of the failure of trust region steps. The numerical results show that the new method may be capable of processing some large-scale problems.

An interior algorithm for nonlinear optimization that combines line search and trust region steps

An interior-point method for nonlinear programming is presented. It enjoys the flexibility of switching between a line search method that computes steps by factoring the primal-dual equations and a trust region method that uses a conjugate gradient iteration. Steps computed by direct factorization are always tried first, but if they are deemed ineffective, a trust region iteration that guarantees progress toward stationarity is invoked. To demonstrate its effectiveness, the algorithm is implemented in the Knitro [6,28] software package and is extensively tested on a wide selection of test problems.

Combining trust region and linesearch algorithm for equality constrained optimization

In this paper, a combining trust region and line search algorithm for equality constrained optimization is proposed. At each iteration, we only need to solve the trust region subproblem once, when the trust region trial step can not be accepted, we switch to line search to obtain the next iteration. Hence, the difficulty of repeated solving trust region subproblem in an iterate is avoided. In order to allow the direction of negative curvature, we add second correction step in trust region step and employ nonmonotone technique in line search. The global convergence and local superlinearly rate are established under certain assumptions. Some numerical examples are given to illustrate the efficiency of the proposed algorithm.

A Modified Trust Region Algorithm

In this article, we propose an algorithm which solves unconstrained optimization problems by combining the trust region method with the quasi-Newton line search method. During consecutive trust region steps, when the quasi-Newton matrix B k approximates the Hessian matrix of the objective function at x k well, the algorithm tries a full quasi-Newton step and, if this step is successful, continues trying quasi-Newton steps. On the other hand, when employing the line search method, if the quasi-Newton direction is nearly orthogonal to the gradient of the objective function or the step size becomes too small, the algorithm switches back to trust region steps. Convergence properties of the algorithm are proved, and numerical results are presented.