Worst-case Iteration Complexity Research Articles

A Warm-Start Strategy in Interior Point Methods for Shrinking Horizon Model Predictive Control With Variable Discretization Step

For optimal control problems with random external disturbances, offline algorithms may not perform well because the state cannot be predicted. To solve the problems online, the model predictive control is frequently used, where a series of subproblems with different initial conditions are solved. These convex subproblems could be solved by using interior point methods after discrete approximations. To obtain a solution as accurately as possible under limited computing resources, the variable discretization step is usually considered. To improve efficiency, warm-start methods are considered, which utilize the similarity of subproblem structures. Specifically, the information obtained by solving earlier subproblems is used to solve subsequent subproblems. However, the discretization steps and the sampling numbers of subproblems may be different, and the initial states vary greatly, resulting in poor performance of existing warm-start strategies for interior point methods. In this paper, we present a new warm-start strategy for interior-point methods. In the strategy, a convex combination of components of the earlier optimal solution is used to construct an interpolation point, and we use the convex combination of the modified interpolation point and the cold-start point to obtain a warm-start point. We prove that the worst-case iteration complexity of our strategy is better than that of the cold-start. In the numerical experiment of fuel-optimal planetary powered-descent guidance problems, our strategy reduces the number of iterations of second-order cone programming by about <inline-formula><tex-math notation="LaTeX">$80\%$</tex-math></inline-formula> with the number of samples available in practical applications.

Alfonso: Matlab Package for Nonsymmetric Conic Optimization

We present alfonso, an open-source Matlab package for solving conic optimization problems over nonsymmetric convex cones. The implementation is based on the authors’ corrected analysis of a method of Skajaa and Ye. It enables optimization over any convex cone as long as a logarithmically homogeneous self-concordant barrier is available for the cone or its dual. This includes many nonsymmetric cones, for example, hyperbolicity cones and their duals (such as sum-of-squares cones), semidefinite and second-order cone representable cones, power cones, and the exponential cone. Besides enabling the solution of problems that cannot be cast as optimization problems over a symmetric cone, algorithms for nonsymmetric conic optimization also offer performance advantages for problems whose symmetric cone programming representation requires a large number of auxiliary variables or has a special structure that can be exploited in the barrier computation. The worst-case iteration complexity of alfonso is the best known for nonsymmetric cone optimization: [Formula: see text] iterations to reach an ε-optimal solution, where ν is the barrier parameter of the barrier function used in the optimization. Alfonso can be interfaced with a Matlab function (supplied by the user) that computes the Hessian of a barrier function for the cone. A simplified interface is also available to optimize over the direct product of cones for which a barrier function has already been built into the software. This interface can be easily extended to include new cones. Both interfaces are illustrated by solving linear programs. The oracle interface and the efficiency of alfonso are also demonstrated using an optimal design of experiments problem in which the tailored barrier computation greatly decreases the solution time compared with using state-of-the-art, off-the-shelf conic optimization software. Summary of Contribution: The paper describes an open-source Matlab package for optimization over nonsymmetric cones. A particularly important feature of this software is that, unlike other conic optimization software, it enables optimization over any convex cone as long as a suitable barrier function is available for the cone or its dual, not limiting the user to a small number of specific cones. Nonsymmetric cones for which such barriers are already known include, for example, hyperbolicity cones and their duals (such as sum-of-squares cones), semidefinite and second-order cone representable cones, power cones, and the exponential cone. Thus, the scope of this software is far larger than most current conic optimization software. This does not come at the price of efficiency, as the worst-case iteration complexity of our algorithm matches the iteration complexity of the most successful interior-point methods for symmetric cones. Besides enabling the solution of problems that cannot be cast as optimization problems over a symmetric cone, our software can also offer performance advantages for problems whose symmetric cone programming representation requires a large number of auxiliary variables or has a special structure that can be exploited in the barrier computation. This is also demonstrated in this paper via an example in which our code significantly outperforms Mosek 9 and SCS 2.

Convergence and Complexity Analysis of a Levenberg–Marquardt Algorithm for Inverse Problems

The Levenberg-Marquardt algorithm is one of the most popular algorithms for finding the solution of nonlinear least squares problems. Across different modified variations of the basic procedure, the algorithm enjoys global convergence, a competitive worst case iteration complexity rate, and a guaranteed rate of local convergence for both zero and nonzero small residual problems, under suitable assumptions. We introduce a novel Levenberg-Marquardt method that matches, simultaneously, the state of the art in all of these convergence properties with a single seamless algorithm. Numerical experiments confirm the theoretical behavior of our proposed algorithm.

Subsampled Nonmonotone Spectral Gradient Methods

Abstract This paper deals with subsampled spectral gradient methods for minimizing finite sums. Subsample function and gradient approximations are employed in order to reduce the overall computational cost of the classical spectral gradient methods. The global convergence is enforced by a nonmonotone line search procedure. Global convergence is proved provided that functions and gradients are approximated with increasing accuracy. R-linear convergence and worst-case iteration complexity is investigated in case of strongly convex objective function. Numerical results on well known binary classification problems are given to show the effectiveness of this framework and analyze the effect of different spectral coefficient approximations arising from the variable sample nature of this procedure.

Primal-dual interior point methods for Semidefinite programming based on a new type of kernel functions

In this paper, we propose the first hyperbolic-logarithmic kernel function for Semidefinite programming problems. By simple analysis tools, several properties of this kernel function are used to compute the total number of iterations. We show that the worst-case iteration complexity of our algorithm for large-update methods improves the obtained iteration bounds based on hyperbolic [24] as well as classic kernel functions. For small-update methods, we derive the best known iteration bound.

Subsampled inexact Newton methods for minimizing large sums of convex functions

Abstract This paper deals with the minimization of a large sum of convex functions by inexact Newton (IN) methods employing subsampled functions, gradients and Hessian approximations. The conjugate gradient method is used to compute the IN step and global convergence is enforced by a nonmonotone line-search procedure. The aim is to obtain methods with affordable costs and fast convergence. Assuming strongly convex functions, R-linear convergence and worst-case iteration complexity of the procedure are investigated when functions and gradients are approximated with increasing accuracy. A set of rules for the forcing parameters and subsample Hessian sizes are derived that ensure local q-linear/q-superlinear convergence of the proposed method. The random choice of the Hessian subsample is also considered and convergence in the mean square, both for finite and infinite sums of functions, is proved. Finally, the analysis of global convergence with asymptotic R-linear rate is extended to the case of the sum of convex functions and strongly convex objective function. Numerical results on well-known binary classification problems are also given. Adaptive strategies for selecting forcing terms and Hessian subsample size, streaming out of the theoretical analysis, are employed and the numerical results show that they yield effective IN methods.

A residual-based algorithm for solving a class of structured nonsmooth optimization problems

In this paper, we consider a class of structured nonsmooth optimization problem in which the first component of the objective is a smooth function while the second component is the sum of one-dimensional nonsmooth functions. We first verify that every minimizer of this problem is a solution of an equation $$h(x)=0$$, where h is continuous but not differentiable, and moreover $$-h(x)$$ is a descent direction of the objective at $$x\in \mathbb {R}^n$$ if $$h(x)\ne 0$$. Then by using $$-h(x)$$ as a search direction, we propose a residual-based algorithm for solving this problem. Under proper conditions, we verify that any accumulation point of the sequence of iterates generated by our algorithm is a first-order stationary point of the problem. Additionally, we prove that the worst-case iteration-complexity for finding an $$\epsilon $$ first-order stationary point is $$O(\epsilon ^{-2})$$. Numerical results have shown the efficiency of this algorithm.

A Single-Phase, Proximal Path-Following Framework

We propose a new proximal path-following framework for a class of constrained convex problems. We consider settings where the nonlinear—and possibly nonsmooth—objective part is endowed with a proximity operator, and the constraint set is equipped with a self-concordant barrier. Our approach relies on the following two main ideas. First, we reparameterize the optimality condition as an auxiliary problem, such that a good initial point is available; by doing so, a family of alternative paths toward the optimum is generated. Second, we combine the proximal operator with path-following ideas to design a single-phase, proximal path-following algorithm. We prove that our algorithm has the same worst-case iteration complexity bounds as in standard path-following methods from the literature but does not require an initial phase. Our framework also allows inexactness in the evaluation of proximal Newton directions, without sacrificing the worst-case iteration complexity. We demonstrate the merits of our algorithm via three numerical examples, where proximal operators play a key role.

Primal interior-point decomposition algorithms for two-stage stochastic extended second-order cone programming

ABSTRACTWe study and solve the two-stage stochastic extended second-order cone programming problem. We show that the barrier recourse functions and the composite barrier functions for this optimization problem are self-concordant families with respect to barrier parameters. These results are used to develop primal decomposition-based interior-point algorithms. The worst case iteration complexity of the developed algorithms is shown to be the same as that for the short- and long-step primal interior algorithms applied to the extensive formulation of our problem.

Primal–dual interior-point method for linear optimization based on a kernel function with trigonometric growth term

In this paper, we propose a large-update primal–dual interior-point algorithm for linear optimization problems based on a new kernel function with a trigonometric growth term. By simple analysis, we prove that in the large neighbourhood of the central path, the worst case iteration complexity of the new algorithm is bounded above by , which matches the currently best known iteration bound for large-update methods. Moreover, we show that, most of the so far proposed kernel functions can be rewritten as a kernel function with trigonometric growth term. Finally, numerical experiments on some test problems confirm that the new kernel function is well promising in practice in comparison with some existing kernel functions in the literature.

An inexact regularized Newton framework with a worst-case iteration complexity of $ {\mathscr O}(\varepsilon^{-3/2}) $ for nonconvex optimization

Abstract An algorithm for solving smooth nonconvex optimization problems is proposed that, in the worst-case, takes ${\mathscr O}(\varepsilon ^{-3/2})$ iterations to drive the norm of the gradient of the objective function below a prescribed positive real number $\varepsilon $ and can take ${\mathscr O}(\varepsilon ^{-3})$ iterations to drive the leftmost eigenvalue of the Hessian of the objective above $-\varepsilon $. The proposed algorithm is a general framework that covers a wide range of techniques including quadratically and cubically regularized Newton methods, such as the Adaptive Regularization using Cubics (arc) method and the recently proposed Trust-Region Algorithm with Contractions and Expansions (trace). The generality of our method is achieved through the introduction of generic conditions that each trial step is required to satisfy, which in particular allows for inexact regularized Newton steps to be used. These conditions center around a new subproblem that can be approximately solved to obtain trial steps that satisfy the conditions. A new instance of the framework, distinct from arc and trace, is described that may be viewed as a hybrid between quadratically and cubically regularized Newton methods. Numerical results demonstrate that our hybrid algorithm outperforms a cubically regularized Newton method.

Self-concordant inclusions: a unified framework for path-following generalized Newton-type algorithms

We study a class of monotone inclusions called “self-concordant inclusion” which covers three fundamental convex optimization formulations as special cases. We develop a new generalized Newton-type framework to solve this inclusion. Our framework subsumes three schemes: full-step, damped-step, and path-following methods as specific instances, while allows one to use inexact computation to form generalized Newton directions. We prove the local quadratic convergence of both full-step and damped-step algorithms. Then, we propose a new two-phase inexact path-following scheme for solving this monotone inclusion which possesses an $${\mathcal {O}}(\sqrt{\nu }\log (1/\varepsilon ))$$-worst-case iteration-complexity to achieve an $$\varepsilon $$-solution, where $$\nu $$ is the barrier parameter and $$\varepsilon $$ is a desired accuracy. As byproducts, we customize our scheme to solve three convex problems: the convex–concave saddle-point problem, the nonsmooth constrained convex program, and the nonsmooth convex program with linear constraints. We also provide three numerical examples to illustrate our theory and compare with existing methods.

Quasi-Newton methods for constrained nonlinear systems: complexity analysis and applications

We address the solution of constrained nonlinear systems by new linesearch quasi-Newton methods. These methods are based on a proper use of the projection map onto the convex constraint set and on a derivative-free and nonmonotone linesearch strategy. The convergence properties of the proposed methods are presented along with a worst-case iteration complexity bound. Several implementations of the proposed scheme are discussed and validated on bound-constrained problems including gas distribution network models. The results reported show that the new methods are very efficient and competitive with an existing affine-scaling procedure.

Complexity Analysis of a Trust Funnel Algorithm for Equality Constrained Optimization

A method is proposed for solving equality constrained nonlinear optimization problems involving twice continuously differentiable functions. The method employs a trust funnel approach consisting of two phases: a first phase to locate an $\epsilon$-feasible point and a second phase to seek optimality while maintaining at least $\epsilon$-feasibility. Two-phase approaches of this kind based on a cubic regularization methodology have recently been proposed along with supporting worst-case iteration complexity analyses. Notably, in these approaches, the objective function is completely ignored in the first phase when $\epsilon$-feasibility is sought. The main contribution of the method proposed in this paper is that the same worst-case iteration complexity is achieved, but with a first phase that also accounts for improvements in the objective function. As such, the method attempts to put significantly less burden on the second phase for seeking optimality.

A unified complexity analysis of interior point methods for semidefinite problems based on trigonometric kernel functions

In this paper, we present an interior-point algorithm for semidefinite optimization (SDO) problems based on a new generic trigonometric kernel function, which is constructed by introducing some new conditions on the kernel function. Based on these conditions, we propose a new trigonometric kernel function and present some properties of this function. We present some complexity results for the generic kernel function and prove that, a large-update primal–dual interior-point method for solving SDO problems with this new kernel function enjoys as worst case iteration complexity bound which matches the currently best known complexity bound for large update methods. Moreover, some numerical results show that the new proposed kernel function has better results than the other trigonometric kernel functions.

On the use of the energy norm in trust-region and adaptive cubic regularization subproblems

We consider solving unconstrained optimization problems by means of two popular globalization techniques: trust-region (TR) algorithms and adaptive regularized framework using cubics (ARC). Both techniques require the solution of a so-called ``subproblem'' in which a trial step is computed by solving an optimization problem involving an approximation of the objective function, called ``the model. The latter is supposed to be adequate in a neighborhood of the current iterate. In this paper, we address an important practical question related with the choice of the norm for defining the neighborhood. More precisely, assuming here that the Hessian $B$ of the model is symmetric positive definite, we propose the use of the so-called ``energy norm'' -- defined by $\|x\|_B= \sqrt{x^TBx}$ for all $x \in \real^n$ -- in both TR and ARC techniques. We show that the use of this norm induces remarkable relations between the trial step of both methods that can be used to obtain efficient practical algorithms. We furthermore consider the use of truncated Krylov subspace methods to obtain an approximate trial step for large scale optimization. Within the energy norm, we obtain line search algorithms along the Newton direction, with a special backtracking strategy and an acceptability condition in the spirit of TR/ARC methods. The new line search algorithm, derived by ARC, enjoys a worst-case iteration complexity of $\mathcal{O}(\epsilon^{-3/2})$. We show the good potential of the energy norm on a set of numerical experiments.

An Efficient Global Algorithm for Single-Group Multicast Beamforming

Consider the single-group multicast beamforming problem, where multiple users receive the same data stream simultaneously from a single transmitter. The problem is NP-hard and all existing algorithms for the problem either find suboptimal approximate or local stationary solutions. In this paper, we propose an efficient branch-and-bound algorithm for the problem that is guaranteed to find its global solution. To the best of our knowledge, our proposed algorithm is the first tailored global algorithm for the single-group multicast beamforming problem. Simulation results show that our proposed algorithm is computationally efficient (albeit its theoretical worst-case iteration complexity is exponential with respect to the number of receivers) and it significantly outperforms a state-of-the-art general-purpose global optimization solver called Baron. Our proposed algorithm provides an important benchmark for performance evaluation of existing algorithms for the same problem. By using it as the benchmark, we show that two state-of-the-art algorithms, semidefinite relaxation algorithm and successive linear approximation algorithm, work well when the problem dimension (i.e., the number of antennas at the transmitter and the number of receivers) is small but their performance deteriorates quickly as the problem dimension increases.

Stochastic Quasi-Newton Methods for Nonconvex Stochastic Optimization

In this paper we study stochastic quasi-Newton methods for nonconvex stochastic optimization, where we assume that noisy information about the gradients of the objective function is available via a stochastic first-order oracle ($\mathcal{SFO}$). We propose a general framework for such methods, for which we prove almost sure convergence to stationary points and analyze its worst-case iteration complexity. When a randomly chosen iterate is returned as the output of such an algorithm, we prove that in the worst case, the $\mathcal{SFO}$-calls complexity is $O(\epsilon^{-2})$ to ensure that the expectation of the squared norm of the gradient is smaller than the given accuracy tolerance $\epsilon$. We also propose a specific algorithm, namely, a stochastic damped limited-memory BFGS (SdLBFGS) method, that falls under the proposed framework. Moreover, we incorporate the stochastic variance reduced gradient variance reduction technique into the proposed SdLBFGS method and analyze its $\mathcal{SFO}$-calls complexity. Numerical results on a nonconvex binary classification problem using a support vector machine and a multiclass classification problem using neural networks are reported.

Adaptive smoothing algorithms for nonsmooth composite convex minimization

We propose an adaptive smoothing algorithm based on Nesterov's smoothing technique in Nesterov (Math Prog 103(1):127---152, 2005) for solving fully nonsmooth composite convex optimization problems. Our method combines both Nesterov's accelerated proximal gradient scheme and a new homotopy strategy for smoothness parameter. By an appropriate choice of smoothing functions, we develop a new algorithm that has the $$\mathcal {O}\left( \frac{1}{\varepsilon }\right) $$O1?-worst-case iteration-complexity while preserves the same complexity-per-iteration as in Nesterov's method and allows one to automatically update the smoothness parameter at each iteration. Then, we customize our algorithm to solve four special cases that cover various applications. We also specify our algorithm to solve constrained convex optimization problems and show its convergence guarantee on a primal sequence of iterates. We demonstrate our algorithm through three numerical examples and compare it with other related algorithms.

A trust region algorithm with a worst-case iteration complexity of $$\mathcal{O}(\epsilon ^{-3/2})$$ O ( ϵ - 3 / 2 ) for nonconvex optimization

We propose a trust region algorithm for solving nonconvex smooth optimization problems. For any $$\overline{\epsilon }\in (0,\infty )$$∈¯?(0,?), the algorithm requires at most $$\mathcal{O}(\epsilon ^{-3/2})$$O(∈-3/2) iterations, function evaluations, and derivative evaluations to drive the norm of the gradient of the objective function below any $$\epsilon \in (0,\overline{\epsilon }]$$∈?(0,∈¯]. This improves upon the $$\mathcal{O}(\epsilon ^{-2})$$O(∈-2) bound known to hold for some other trust region algorithms and matches the $$\mathcal{O}(\epsilon ^{-3/2})$$O(∈-3/2) bound for the recently proposed Adaptive Regularisation framework using Cubics, also known as the arc algorithm. Our algorithm, entitled trace, follows a trust region framework, but employs modified step acceptance criteria and a novel trust region update mechanism that allow the algorithm to achieve such a worst-case global complexity bound. Importantly, we prove that our algorithm also attains global and fast local convergence guarantees under similar assumptions as for other trust region algorithms. We also prove a worst-case upper bound on the number of iterations, function evaluations, and derivative evaluations that the algorithm requires to obtain an approximate second-order stationary point.