Worst-case Iteration Complexity Research Articles

A large-update primal–dual interior-point algorithm for second-order cone optimization based on a new proximity function

In this paper, we propose a large-update primal–dual interior-point algorithm for second-order cone optimization (SOCO) based on a class of kernel functions consisting of a trigonometric barrier term. The algorithm starts from a strictly feasible point and generates a sequence of points converging to an optimal solution of the problem. Using a simple analysis, we show that the algorithm has worst case iteration complexity for large-update primal–dual interior point methods which coincides with the so far best-known iteration bound for SOCO.

A Barzilai–Borwein-Like Iterative Half Thresholding Algorithm for the $$L_{1/2}$$ L 1 / 2 Regularized Problem

In this paper, we propose a Barzilai---Borwein-like iterative half thresholding algorithm for the $$L_{1/2}$$L1/2 regularized problem. The algorithm is closely related to the iterative reweighted minimization algorithm and the iterative half thresholding algorithm. Under mild conditions, we verify that any accumulation point of the sequence of iterates generated by the algorithm is a first-order stationary point of the $$L_{1/2}$$L1/2 regularized problem. We also prove that any accumulation point is a local minimizer of the $$L_{1/2}$$L1/2 regularized problem when additional conditions are satisfied. Furthermore, we show that the worst-case iteration complexity for finding an $$\varepsilon $$? scaled first-order stationary point is $$O(\varepsilon ^{-2})$$O(?-2). Preliminary numerical results show that the proposed algorithm is practically effective.

A smoothing SQP framework for a class of composite $$L_q$$ L q minimization over polyhedron

The composite $$L_q~(0<q<1)$$ minimization problem over a general polyhedron has received various applications in machine learning, wireless communications, image restoration, signal reconstruction, etc. This paper aims to provide a theoretical study on this problem. First, we derive the Karush–Kuhn–Tucker (KKT) optimality conditions for local minimizers of the problem. Second, we propose a smoothing sequential quadratic programming framework for solving this problem. The framework requires a (approximate) solution of a convex quadratic program at each iteration. Finally, we analyze the worst-case iteration complexity of the framework for returning an $$\epsilon $$ -KKT point; i.e., a feasible point that satisfies a perturbed version of the derived KKT optimality conditions. To the best of our knowledge, the proposed framework is the first one with a worst-case iteration complexity guarantee for solving composite $$L_q$$ minimization over a general polyhedron.

A second-order method for strongly convex $$\ell _1$$ ℓ 1 -regularization problems

In this paper a robust second-order method is developed for the solution of strongly convex $$\ell _1$$l1-regularized problems. The main aim is to make the proposed method as inexpensive as possible, while even difficult problems can be efficiently solved. The proposed approach is a primal-dual Newton conjugate gradients (pdNCG) method. Convergence properties of pdNCG are studied and worst-case iteration complexity is established. Numerical results are presented on synthetic sparse least-squares problems and real world machine learning problems.

An interior-point method for $$P_*(\kappa )$$ P ∗ ( κ ) -linear complementarity problem based on a trigonometric kernel function

Recently, El Ghami (Optim Theory Decis Mak Oper Res Appl 31:331–349, 2013) proposed a primal dual interior point method for $$P_*(\kappa )$$ -Linear Complementarity Problem (LCP) based on a trigonometric barrier term and obtained the worst case iteration complexity as $$O\left( (1+2\kappa )n^{\frac{3}{4}}\log \frac{n}{\epsilon }\right) $$ for large-update methods. In this paper, we present a large update primal–dual interior point algorithm for $$P_{*}(\kappa )$$ -LCP based on a new trigonometric kernel function. By a simple analysis, we show that our algorithm based on the new kernel function enjoys the worst case $$O\left( (1+2\kappa )\sqrt{n}\log n\log \frac{n}{\epsilon }\right) $$ iteration bound for solving $$P_*(\kappa )$$ -LCP. This result improves the worst case iteration bound obtained by El Ghami for $$P_*(\kappa )$$ -LCP based on trigonometric kernel functions significantly.

Complexity analysis of interior point algorithms for non-Lipschitz and nonconvex minimization

We propose a first order interior point algorithm for a class of non-Lipschitz and nonconvex minimization problems with box constraints, which arise from applications in variable selection and regularized optimization. The objective functions of these problems are continuously differentiable typically at interior points of the feasible set. Our first order algorithm is easy to implement and the objective function value is reduced monotonically along the iteration points. We show that the worst-case iteration complexity for finding an $$\epsilon $$ ∈ scaled first order stationary point is $$O(\epsilon ^{-2})$$ O ( ∈ - 2 ) . Furthermore, we develop a second order interior point algorithm using the Hessian matrix, and solve a quadratic program with a ball constraint at each iteration. Although the second order interior point algorithm costs more computational time than that of the first order algorithm in each iteration, its worst-case iteration complexity for finding an $$\epsilon $$ ∈ scaled second order stationary point is reduced to $$O(\epsilon ^{-3/2})$$ O ( ∈ - 3 / 2 ) . Note that an $$\epsilon $$ ∈ scaled second order stationary point must also be an $$\epsilon $$ ∈ scaled first order stationary point.

On the Iteration Complexity of Cyclic Coordinate Gradient Descent Methods

The cyclic (block) coordinate gradient descent method is an optimization method that has attracted much interest in applications of applied mathematics, statistics, and engineering. Reasons for this include its simplicity, speed, and stability, as well as its competitive performance on separable nonsmooth convex minimization problems, whose objective function is the sum of a smooth function and a separable (and possibly nonsmooth) convex function, such as the $\ell_1$-regularized linear least squares problem and the $\ell_1$-regularized logistic regression problem. But very little is known about the worst case iteration complexity of the method for solving the separable nonsmooth convex minimization problem. We prove that the method terminates in $O(1/\epsilon)$ iterations with an $\epsilon$-optimal solution, or equivalently, the convergence rate for it is $O(1/k)$, where $k$ is the iteration counter, when the smooth function of the objective has a Lipschitz gradient. Also, the linear rate convergence of the method is proved when the objective is a strongly convex function having a Lipschitz gradient or when the smooth function of the objective is a composition of a strong convex function having a Lipschitz gradient with a linear function, the convex function of the objective is polyhedral, and there is a real number whose corresponding level set of the convex function contains the set of optimal solutions and is bounded.

An Inexact Accelerated Proximal Gradient Method for Large Scale Linearly Constrained Convex SDP

The accelerated proximal gradient (APG) method, first proposed by Nesterov for minimizing smooth convex functions, later extended by Beck and Teboulle to composite convex objective functions, and studied in a unifying manner by Tseng, has proven to be highly efficient in solving some classes of large scale structured convex optimization (possibly nonsmooth) problems, including nuclear norm minimization problems in matrix completion and $l_1$ minimization problems in compressed sensing. The method has superior worst-case iteration complexity over the classical projected gradient method and usually has good practical performance on problems with appropriate structures. In this paper, we extend the APG method to the inexact setting, where the subproblem in each iteration is solved only approximately, and show that it enjoys the same worst-case iteration complexity as the exact counterpart if the subproblems are progressively solved to sufficient accuracy. We apply our inexact APG method to solve large scale convex quadratic semidefinite programming (QSDP) problems of the form $\min\{ \frac{1}{2}\langle x,\,\mathcal{Q}(x)\rangle + \langle c,\,x\rangle\mid \mathcal{A}(x) = b, x\succeq 0\}$, where $\mathcal{Q},\mathcal{A}$ are given linear maps and $b,c$ are given data. The subproblem in each iteration is solved by a semismooth Newton-CG (SSNCG) method with warm-start using the iterate from the previous iteration. Our APG-SSNCG method is demonstrated to be efficient for QSDP problems whose positive semidefinite linear maps $\mathcal{Q}$ are highly ill-conditioned or rank deficient.

Self-concordance and Decomposition-based Interior Point Methods for the Two-stage Stochastic Convex Optimization Problem

We study the two-stage stochastic convex optimization problem whose first- and second-stage feasible regions admit a self-concordant barrier. We show that the barrier recourse functions and the composite barrier functions for this problem form self-concordant families. These results are used to develop prototype primal interior point decomposition algorithms that are more suitable for a heterogeneous distributed computing environment. We show that the worst case iteration complexity of the proposed algorithms is the same as that for the short- and long-step primal interior algorithms applied to the extensive formulation of this problem. The generality of our results allows the possibility of using barriers other than the standard log-barrier in an algorithmic framework.

Counterexample to a Conjecture on an Infeasible Interior-Point Method

In [SIAM J. Optim., 16 (2006), pp. 1110–1136], Roos proved that the devised full-step infeasible algorithm has $O(n)$ worst-case iteration complexity. This complexity bound depends linearly on a parameter $\bar{\kappa}(\zeta)$, which is proved to be less than $\sqrt{2n}$. Based on extensive computational evidence (hundreds of thousands of randomly generated problems), Roos conjectured that $\bar{\kappa}(\zeta)=1$ (Conjecture 5.1 in the above-mentioned paper), which would yield an $O(\sqrt{n})$ iteration full-Newton step infeasible interior-point algorithm. In this paper we present an example showing that $\bar{\kappa}(\zeta)$ is in the order of $\sqrt{n}$, the same order as that proved in Roos's original paper. In other words, the conjecture is false.

Convergence of a Weighted Barrier Decomposition Algorithm for Two-Stage Stochastic Programming with Discrete Support

Mehrotra and Özevin [SIAM J. Optim., 19 (2009), pp. 1846–1880] computationally found that a weighted primal barrier decomposition algorithm significantly outperforms the equally weighted barrier decomposition proposed and analyzed in [G. Zhao, Math. Program., 90 (2001), pp. 507–536; S. Mehrotra and M. G. Özevin, Oper. Res., 57 (2009), pp. 964–974; S. Mehrotra and M. G. Özevin, SIAM J. Optim., 18 (2007), pp. 206–222]. Here we consider a weighted barrier that allows us to analyze iteration complexity of algorithms in all of the aforementioned publications in a unified framework. In particular, we prove self-concordance parameter values for the weighted barrier and using these values give a worst-case iteration complexity bound for the weighted decomposition algorithm.

A HYBRID ADAPTIVE ALGORITHM FOR LINEAR OPTIMIZATION

Recently, using the framework of self-regularity, Salahi in his Ph.D. thesis proposed an adaptive single step algorithm which takes advantage of the current iterate information to find an appropriate barrier parameter rather than using a fixed fraction of the current duality gap. However, his algorithm might do at most one bad step after each good step in order to keep the iterate in a certain neighborhood of the central path. In this paper, using the same framework, we propose a hybrid adaptive algorithm. Depending on the position of the current iterate, our new algorithm uses either the classical Newton search direction or a self-regular search direction. The larger the distance from the central path, the larger the barrier degree of the self-regular search direction is. Unlike the classical approach, here we control the iterates by guaranteeing certain reduction of the proximity measure. This itself leads to a one dimensional equation which determines the target barrier parameter at each iteration. This allows us to have a large update algorithm without any need for safeguard or special steps. Finally, we prove that our hybrid adaptive algorithm has an [Formula: see text] worst case iteration complexity.

A Polynomial-Time Interior-Point Method for Conic Optimization, With Inexact Barrier Evaluations

We consider a primal-dual short-step interior-point method for conic convex optimization problems for which exact evaluation of the gradient and Hessian of the primal and dual barrier functions is either impossible or prohibitively expensive. As our main contribution, we show that if approximate gradients and Hessians of the primal barrier function can be computed, and the relative errors in such quantities are not too large, then the method has polynomial worst-case iteration complexity. (In particular, polynomial iteration complexity ensues when the gradient and Hessian are evaluated exactly.) In addition, the algorithm requires no evaluation—or even approximate evaluation—of quantities related to the barrier function for the dual cone, even for problems in which the underlying cone is not self-dual.

Mehrotra-type predictor-corrector algorithm revisited

Motivated by a numerical example that shows that a feasible version of Mehrotra's original predictor-corrector algorithm might be inefficient in practice, Salahi et al. [M. Salahi, J. Peng and T. Terlaky, On Mehrotra-type predictor-corrector algorithms, to apper in SIAM J. Optim.] proposed a so-called safeguard-based variant of the algorithm that enjoys polynomial iteration complexity, although its practical efficiency is preserved. In this paper, we analyse the same Mehrotra's algorithm from a different perspective. We give a condition on the maximum step size in the predictor direction, the violation of which might imply a very small or zero step size in the corrector direction of the algorithm. This might explain the reason for occasional ill behaviour of the feasible version of Mehrotra's original algorithm. We propose to cut the maximum step size in the predictor direction if it is above a certain threshold. If this cut does not give a desirable step size, then we cut it for the second time that allows us to give a lower bound for the step size in the corrector direction. This enables us to prove an 𝒪(n 5/2log (n/ε)) worst case iteration complexity bound for the new algorithm. By slightly modifying the Newton system in the corrector step, we reduce the iteration complexity to 𝒪 (n 3/2log (n/ε)). Finally, we report some illustrative computational results using the McIPM software package.

Postponing the choice of the barrier parameter in Mehrotra-type predictor–corrector algorithms

Recently Salahi et al. have considered a variant of Mehrotra’s celebrated predictor–corrector algorithm. By a numerical example they showed that this variant might make very small steps in order to keep the iterate in a certain neighborhood of the central path, that itself implies the inefficiency of the algorithm. This observation motivated them to incorporate a safeguard in their algorithmic scheme that gives a lower bound for the step size at each iteration and thus imply polynomial iteration complexity. In this paper we propose a different approach that enables us to have control on the iterates. Our new approach is based on postponing the choice of the barrier parameter and it does not require any safeguard strategy. To do so, first we fix a step size in the corrector step, then by solving a one dimensional optimization problem we calculate the barrier parameter. Finally, using the calculated barrier parameter we make the next step. It is proved for the feasible case that our new algorithm stops after at most O n 2 log n ϵ iterations without any safeguard strategy. We further modified the proposed algorithm by slightly changing the Newton system in the corrector step. This variant enjoys an even better worst case iteration complexity bound, i.e., O n log n ϵ . Superlinear convergence of both algorithms are established. Finally, our numerical results demonstrate that a theoretically rigorous IPM may have the same, or better computational efficiency than the ones enhanced by aggressive heuristics.

A Constant-Potential Infeasible-Start Interior-Point Algorithmwith Computational Experiments and Applications

We present a constant-potential infeasible-start interior-point (INFCP) algorithm for linear programming (LP) problems with a worst-case iteration complexity analysis as well as some computational results.The performance of the INFCP algorithm is compared to those of practical interior-point algorithms. New features of the algorithm include a heuristic method for computing a “good” starting point and a procedure for solving the augmented system arising from stochastic programming with simple recourse. We also present an application to large scale planning problems under uncertainty.