Trust Region Problem Research Articles

The mixed integer trust region problem

The mixed integer trust region problem

On the exactness of a simple relaxation for the extended Celis–Dennis–Tapia subproblem

In this work, we study the so-called extended Celis–Dennis–Tapia subproblem that are classical trust region subproblems with an additional quadratic constraint. For this class of problems, we propose a family of relaxations, and study some of its key properties, for instance, convexity issues, exactness of the relaxation, sufficient condition for exactness. The relaxation proposed is obtained adapting to our case a methodology recently developed for trust region problems with an additional conic constraint. We also find sufficient conditions for exactness of the proposed relaxation and discuss the relations between them. Finally, some numerical experiments are presented for testing the proposed relaxation and the sufficient conditions for its tightness.

Optimal Diffusion Learning Over Networks—Part II: Multitask Algorithms

In Part I of this presentation, we have formulated single and multitask quadratic optimization problems, where agents are subject to quadratic, smoothing constraints over a graph. We have focused particularly on single task designs, whereby node uncertainties and their strength relative to un-regularized cost are tackled altogether by means of an adaptive penalty function. In this sequel, we readdress the multitask problem and propose new distributed implementations for their corresponding exact <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">leaky</i> -RLS solutions. We motivate a network formulation from a standalone viewpoint by capitalizing on the fact that 1) for regressors having uncorrelated entries, the performance of an efficient <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">${\mathcal O}(M^2)$</tex-math></inline-formula> conjugate-gradient (CG) realization of the leaky LS solution is identical to the one of an RLS filter; 2) a CG implementation does not require inversion of the underlying sample covariance matrix. Simple arguments yield an extended network-CG algorithm that relies on node-level recursions employing distinct step-sizes. Unlike the exponentially-weighted RLS algorithm, which tapers off regularization over time, a persistent penalty strength conforms with the very purpose of the equivalent network trust-region problem, while granting a well-conditioned solution. The approach further yields another family of single-task algorithms in terms of network linearly constrained solutions, which can be contrasted to the ones proposed in Part I. In particular, the exact linearly-constrained network LMS implementation proposed here outperforms the adaptive relative-variance NLMS, under much lower computational requirements.

A SOCP relaxation based branch-and-bound method for generalized trust-region subproblem

<p style='text-indent:20px;'>This paper proposes a second-order cone programming (SOCP) relaxation for the generalized trust-region problem by exploiting the property that any symmetric matrix and identity matrix can be simultaneously diagonalizable. We show that our proposed SOCP relaxation can provide a lower bound as tight as that of the standard semidefinite programming (SDP) relaxation. Moreover, we provide a sufficient condition under which the proposed SOCP relaxation is exact. Since the standard SDP relaxation suffers from a much heavier computing burden, the proposed SOCP relaxation has a much higher efficiency in solving process. Then we design a branch-and-bound algorithm based on this SOCP relaxation to obtain the global optimal solution for a general problem. Three types of numerical experiments are carried out to demonstrate the effectiveness and efficiency of our proposed SOCP relaxation.

Trlib: a vector-free implementation of the GLTR method for iterative solution of the trust region problem

We describe trlib, a library that implements a variant of Generalized Lanczos method [Gould et al. in SIAM J. Opt. 9(2), 504–525, 1999] for solving the trust region problem. Our implementation has several distinct features that set it apart from preexisting ones. We implement both conjugate gradient (CG) and Lanczos iterations for assembly of Krylov subspaces. A vector- and matrix-free reverse communication interface allows the use of most general data structures, such as those arising after discretization of function space problems. The hard case of the trust region problem frequently arises in sequential methods for nonlinear optimization. In this implementation, we made an effort to fully address the hard case in an exact way by considering all invariant Krylov subspaces. We investigate the numerical performance of trlib on the full subset of unconstrained problems of the CUTEst benchmark set. In addition to this, interfacing the PDE discretization toolkit FEniCS with trlib using the vector-free reverse communication interface is demonstrated for a family of PDE-constrained control trust region problems adapted from the OPTPDE collection.

Extended trust-region problems with one or two balls: exact copositive and Lagrangian relaxations

We establish a geometric condition guaranteeing exact copositive relaxation for the nonconvex quadratic optimization problem under two quadratic and several linear constraints, and present sufficient conditions for global optimality in terms of generalized Karush–Kuhn–Tucker multipliers. The copositive relaxation is tighter than the usual Lagrangian relaxation. We illustrate this by providing a whole class of quadratic optimization problems that enjoys exactness of copositive relaxation while the usual Lagrangian duality gap is infinite. Finally, we also provide verifiable conditions under which both the usual Lagrangian relaxation and the copositive relaxation are exact for an extended CDT (two-ball trust-region) problem. Importantly, the sufficient conditions can be verified by solving linear optimization problems.

Computing quasisolutions of nonlinear inverse problems via efficient minimization of trust region problems

Abstract In this paper we present a method for the regularized solution of nonlinear inverse problems, based on Ivanov regularization (also called method of quasi solutions or constrained least squares regularization). This leads to the minimization of a nonconvex cost function under a norm constraint, where nonconvexity is caused by nonlinearity of the inverse problem. Minimization is done by iterative approximation, using (nonconvex) quadratic Taylor expansions of the cost function. This leads to repeated solution of quadratic trust region subproblems with possibly indefinite Hessian. Thus, the key step of the method consists in application of an efficient method for solving such quadratic subproblems, developed by Rendl and Wolkowicz [10]. We here present a convergence analysis of the overall method as well as numerical experiments.

Exactness conditions for an SDP relaxation of the extended trust region problem

In this paper we show that a standard SDP relaxation for so called extended trust-region problem is equivalent to a convex quadratic problem, with a linear objective and constraint functions and some additional simple convex quadratic constraints. Through this equivalence, new conditions, generalizing the ones existing in the literature, under which the SDP relaxation is exact, are introduced.

Copositivity for second-order optimality conditions in general smooth optimization problems

Second-order local optimality conditions involving copositivity of the Hessian of the Lagrangian on the reduced linearization cone have the advantage that there is only a small gap between sufficient (the Hessian is strictly copositive) and necessary (the Hessian is copositive) conditions. In this respect, this is a proper generalization of convexity of the Lagrangian. We also specify a copositivity-based variant which is sufficient for global optimality. For (non-convex) quadratic optimization problems over polyhedra (QPs), the distinction between sufficiency and necessity vanishes, both for local and global optimality. However, in the strictly copositive case we can provide a distance lower (error) bound of the increment around a local minimizer . This is a refinement of an earlier result which focussed on mere (non-strict) copositivity. In addition, an apparently new variant of constraint qualification (CQ) is presented which is implied by Abadie’s CQ and which is suitable for second-order analysis. This new reflected Abadie CQ is neither implied, nor implies, Guignard’s CQ. However, it implies the necessary second-order local optimality condition based on copositivity. Application to the trust-region problem and several (counter) examples illustrates the advantage of this approach.

A linear-time algorithm for trust region problems

We consider the fundamental problem of minimizing a general quadratic function over an ellipsoidal domain, also known as the trust region (sub)problem. We give the first provable linear-time (in the number of non-zero entries of the input) algorithm for approximately solving this problem. Specifically, our algorithm returns an $$\epsilon $$∈-approximate solution in runtime of order N/$$\sqrt{\epsilon }$$∈, where N is the number of non-zero entries in the input. This matches the runtime of Nesterov's accelerated gradient descent, suitable for the special case in which the quadratic objective is convex, and the runtime of the Lanczos method which is applicable when the problem is purely quadratic.

Generalized Farkas’ lemma and gap-free duality for minimax DC optimization with polynomials and robust quadratic optimization

Motivated by robust (non-convex) quadratic optimization over convex quadratic constraints, in this paper, we examine minimax difference of convex (dc) optimization over convex polynomial inequalities. By way of generalizing the celebrated Farkas' lemma to inequality systems involving the maximum of dc functions and convex polynomials, we show that there is no duality gap between a minimax DC polynomial program and its associated conjugate dual problem. We then obtain strong duality under a constraint qualification. Consequently, we present characterizations of robust solutions of uncertain general non-convex quadratic optimization problems with convex quadratic constraints, including uncertain trust-region problems.

Narrowing the difficulty gap for the Celis–Dennis–Tapia problem

We study the Celis–Dennis–Tapia (CDT) problem: minimize a non-convex quadratic function over the intersection of two ellipsoids. In contrast to the well-studied trust region problem where the feasible set is just one ellipsoid, the CDT problem is not yet fully understood. Our main objective in this paper is to narrow the difficulty gap that occurs when the Hessian of the Lagrangian is indefinite at all Karush–Kuhn–Tucker points. We prove new sufficient and necessary conditions both for local and global optimality, based on copositivity, giving a complete characterization in the degenerate case.

Trust-region problems with linear inequality constraints: exact SDP relaxation, global optimality and robust optimization

The trust-region problem, which minimizes a nonconvex quadratic function over a ball, is a key subproblem in trust-region methods for solving nonlinear optimization problems. It enjoys many attractive properties such as an exact semi-definite linear programming relaxation (SDP-relaxation) and strong duality. Unfortunately, such properties do not, in general, hold for an extended trust-region problem having extra linear constraints. This paper shows that two useful and powerful features of the classical trust-region problem continue to hold for an extended trust-region problem with linear inequality constraints under a new dimension condition. First, we establish that the class of extended trust-region problems has an exact SDP-relaxation, which holds without the Slater constraint qualification. This is achieved by proving that a system of quadratic and affine functions involved in the model satisfies a range-convexity whenever the dimension condition is fulfilled. Second, we show that the dimension condition together with the Slater condition ensures that a set of combined first and second-order Lagrange multiplier conditions is necessary and sufficient for global optimality of the extended trust-region problem and consequently for strong duality. Through simple examples we also provide an insightful account of our development from SDP-relaxation to strong duality. Finally, we show that the dimension condition is easily satisfied for the extended trust-region model that arises from the reformulation of a robust least squares problem (LSP) as well as a robust second order cone programming model problem (SOCP) as an equivalent semi-definite linear programming problem. This leads us to conclude that, under mild assumptions, solving a robust LSP or SOCP under matrix-norm uncertainty or polyhedral uncertainty is equivalent to solving a semi-definite linear programming problem and so, their solutions can be validated in polynomial time.

An interior point trust region method for nonnegative matrix factorization

Recently, Nonnegative Matrix Factorization (NMF) is a developed method for dimension reduction, feature extraction and data mining, etc. In this paper, we propose an interior point trust region (IPTR) method, which can find a better solution in global region for NMF with general cost functions. First, to control the growth in the size of the solution with noise and regularize the solution in iterations, two auxiliary constraints are added into NMF. Then we introduce the logarithmic barrier function to eliminate the nonnegative regularization, and obtain an equivalent quadratic trust region problem by some mathematical calculation. According to the necessary and sufficient conditions of the trust region problem, we obtain a solution of the original problem by solving a parameterized linear system. We apply this method into NMF with different cost functions, including α-divergence, β-divergence, KL-divergence, dual KL(DKL)-divergence, where different cost functions are imposed on different types of data. Numerical experiments demonstrate the high performance of the proposed method.

On the Optimal Value Function of a Trust-Region Problem with Two Constraints

On the Optimal Value Function of a Trust-Region Problem with Two Constraints

Trust Region-Based Optimization for Maximum Mutual Information Estimation of HMMs in Speech Recognition

In this paper, we have proposed two novel optimization methods for discriminative training (DT) of hidden Markov models (HMMs) in speech recognition based on an efficient global optimization algorithm used to solve the so-called trust region (TR) problem, where a quadratic function is minimized under a spherical constraint. In the first method, maximum mutual information estimation (MMIE) of Gaussian mixture HMMs is formulated as a standard TR problem so that the efficient global optimization method can be used in each iteration to maximize the auxiliary function of discriminative training for speech recognition. In the second method, we propose to construct a new auxiliary function for DT of HMMs by adding a quadratic penalty term. The new auxiliary function is constructed to serve as first-order approximation as well as lower bound of the original discriminative objective function within a locality constraint. Due to the lower-bound property, the found optimal point of the new auxiliary function is guaranteed to improve the original discriminative objective function until it converges to a local optimum or stationary point of the objective function. Both TR-based optimization methods have been investigated on two standard large-vocabulary continuous speech recognition tasks, using the WSJ0 and Switchboard databases. Experimental results have shown that the proposed TR methods outperform the conventional EBW method in terms of convergence behavior as well as recognition performance.

A Second Order Approximation Technique for Robust Shape Optimization

We present a second order approximation for the robust counterpart of general uncertain nonlinear programs with state equation given by a partial di erential equation.We show how the approximated worst-case functions, which are the essential part of the approximated robust counterpart, can be formulated as trust-region problems that can be solved effciently using adjoint techniques. Further, we describe how the gradients of the worst-case functions can be computed analytically combining a sensitivity and an adjoint approach. This methodis applied to shape optimization in structural mechanics in order to obtain optimal solutions that are robust with respect to uncertainty in acting forces. Numerical results are presented.

A Note on Wedge Trust Region Radius Update

Wedge trust region method based on traditional trust region is designed for derivative free optimization problems. This method adds a constraint to the trust region problem, which is called “wedge method”. The problem is that the updating strategy of wedge trust region radius is somewhat simple. In this paper, we develop and combine a new radius updating rule with this method. For most test problems, the number of function evaluations is reduced significantly. The experiments demonstrate the effectiveness of the improvement through our algorithm.

Block relaxation and majorization methods for the nearest correlation matrix with factor structure

We propose two numerical methods, namely the alternating block relaxation method and the alternating majorization method, for the problem of nearest correlation matrix with factor structure, which is highly nonconvex. In the block relaxation method, the subproblem is of the standard trust region problem, which is solved by Steighaug's truncated conjugate gradient method or by the exact trust region method. In the majorization method, the subproblem has a closed-form solution. We then apply the majorization method to the case where nonnegative factors are required. The numerical results confirm that the proposed methods work quite well and are competitive against the best available methods.

Regularized Lagrangian duality for linearly constrained quadratic optimization and trust-region problems

In this paper we first establish a Lagrange multiplier condition characterizing a regularized Lagrangian duality for quadratic minimization problems with finitely many linear equality and quadratic inequality constraints, where the linear constraints are not relaxed in the regularized Lagrangian dual. In particular, in the case of a quadratic optimization problem with a single quadratic inequality constraint such as the linearly constrained trust-region problems, we show that the Slater constraint qualification (SCQ) is necessary and sufficient for the regularized Lagrangian duality in the sense that the regularized duality holds for each quadratic objective function over the constraints if and only if (SCQ) holds. A new theorem of the alternative for systems involving both equality constraints and two quadratic inequality constraints plays a key role. We also provide classes of quadratic programs, including a class of CDT-subproblems with linear equality constraints, where (SCQ) ensures regularized Lagrangian duality.