Self-concordant Functions Research Articles

Scalable Frank–Wolfe on Generalized Self-Concordant Functions via Simple Steps

Scalable Frank–Wolfe on Generalized Self-Concordant Functions via Simple Steps

Unified Approach of Interior-Point Algorithms for P∗(κ)\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$P_*(\\kappa )$$\\end{document}-LCPs Using a New Class of Algebraically Equivalent Transformations

We propose new short-step interior-point algorithms (IPAs) for solving P∗(κ)\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$P_*(\\kappa )$$\\end{document}-linear complementarity problems (LCPs). In order to define the search directions, we use the algebraic equivalent transformation (AET) technique of the system describing the central path. A novelty of the paper is that we introduce a whole, new class of AET functions for which a unified complexity analysis of the IPAs is presented. This class of functions differs from the ones used in the literature for determining search directions, like the class of concave functions determined by Haddou, Migot and Omer, self-regular functions, eligible kernel and self-concordant functions. We prove that the IPAs using any member φ\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\varphi $$\\end{document} of the new class of AET functions have polynomial iteration complexity in the size of the problem, in starting point’s duality gap, in the accuracy parameter and in the parameter κ\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\kappa $$\\end{document}.

Some Aspects on a Special Type of $(\alpha,\beta )$-metric

The aim of this paper is twofold. Firstly, we will investigate the link between the condition for the functions $\phi(s)$ from $(\alpha, \beta)$-metrics of Douglas type to be self-concordant and k-self concordant, and the other objective of the paper will be to continue to investigate the recently new introduced $(\alpha, \beta)$-metric ([17]): $$ F(\alpha,\beta)=\frac{\beta^{2}}{\alpha}+\beta+a \alpha $$ where $\alpha=\sqrt{a_{ij}y^{i}y^{j}}$ is a Riemannian metric; $\beta=b_{i}y^{i}$ is a 1-form, and $a\in \left(\frac{1}{4},+\infty\right)$ is a real positive scalar. This kind of metric can be expressed as follows: $F(\alpha,\beta)=\alpha\cdot \phi(s)$, where $\phi(s)=s^{2}+s+a$. In this paper we will study some important results in respect with the above mentioned $(\alpha, \beta)$-metric such as: the Kropina change for this metric, the Main Scalar for this metric and also we will analyze how the condition to be self-concordant and k-self-concordant for the function $\phi(s)$, can be linked with the condition for the metric $F$ to be of Douglas type. self-concordant functions, Kropina change, main scalar.

Set-Limited Functions and Polynomial-Time Interior-Point Methods

In this paper, we revisit some elements of the theory of self-concordant functions. We replace the notion of self-concordant barrier by a new notion of set-limited function, which forms a wider class. We show that the proper set-limited functions ensure polynomial time complexity of the corresponding path-following method (PFM). Our new PFM follows a deviated path, which connects an arbitrary feasible point with the solution of the problem. We present some applications of our approach to the problems of unconstrained optimization, for which it ensures a global linear rate of convergence even in for nonsmooth objective function.

Generalized self-concordant analysis of Frank–Wolfe algorithms

Projection-free optimization via different variants of the Frank–Wolfe method has become one of the cornerstones of large scale optimization for machine learning and computational statistics. Numerous applications within these fields involve the minimization of functions with self-concordance like properties. Such generalized self-concordant functions do not necessarily feature a Lipschitz continuous gradient, nor are they strongly convex, making them a challenging class of functions for first-order methods. Indeed, in a number of applications, such as inverse covariance estimation or distance-weighted discrimination problems in binary classification, the loss is given by a generalized self-concordant function having potentially unbounded curvature. For such problems projection-free minimization methods have no theoretical convergence guarantee. This paper closes this apparent gap in the literature by developing provably convergent Frank–Wolfe algorithms with standard mathcal {O}(1/k) convergence rate guarantees. Based on these new insights, we show how these sublinearly convergent methods can be accelerated to yield linearly convergent projection-free methods, by either relying on the availability of a local liner minimization oracle, or a suitable modification of the away-step Frank–Wolfe method.

A Newton Frank–Wolfe method for constrained self-concordant minimization

We develop a new Newton Frank–Wolfe algorithm to solve a class of constrained self-concordant minimization problems using linear minimization oracles (LMO). Unlike L-smooth convex functions, where the Lipschitz continuity of the objective gradient holds globally, the class of self-concordant functions only has local bounds, making it difficult to estimate the number of linear minimization oracle (LMO) calls for the underlying optimization algorithm. Fortunately, we can still prove that the number of LMO calls of our method is nearly the same as that of the standard Frank-Wolfe method in the L-smooth case. Specifically, our method requires at most $${\mathcal {O}}\big (\varepsilon ^{-(1 + \nu )}\big )$$ LMO’s, where $$\varepsilon $$ is the desired accuracy, and $$\nu \in (0, 0.139)$$ is a given constant depending on the chosen initial point of the proposed algorithm. Our intensive numerical experiments on three applications: portfolio design with the competitive ratio, D-optimal experimental design, and logistic regression with elastic-net regularizer, show that the proposed Newton Frank–Wolfe method outperforms different state-of-the-art competitors.

Optimal step length for the Newton method: case of self-concordant functions

The theoretical foundation of path-following methods is the performance analysis of the (damped) Newton step on the class of self-concordant functions. However, the bounds available in the literature and used in the design of path-following methods are not optimal. In this contribution we use methods of optimal control theory to compute the optimal step length of the Newton method on the class of self-concordant functions, as a function of the initial Newton decrement, and the resulting worst-case decrease of the decrement. The exact bounds are expressed in terms of solutions of ordinary differential equations which cannot be integrated explicitly. We provide approximate numerical and analytic expressions which are accurate enough for use in optimization methods. Consequently, the neighbourhood of the central path in which the iterates of path-following methods are required to stay can be enlarged, enabling faster progress along the central path during each iteration and hence fewer iterations to achieve a given accuracy.

WITHDRAWN: Logarithmic barrier methods and linear connections on manifolds

This article presented several main stages of timeline of evolutionary history of logarithmic barrier functions along with interior point methods and affine connections on manifolds. The crucial idea is many optimization problems can be better stated on manifolds rather than Euclidean space, such as interior point methods, which in turns based on self-concordant functions (logarithmic barrier functions). This led us to the truth, that there is a logical evidence to study the relationships between self-concordant functions and Riemannian manifolds including affine connections.

Greedy Quasi-Newton Methods with Explicit Superlinear Convergence

In this paper, we study greedy variants of quasi-Newton methods. They are based on the updating formulas from a certain subclass of the Broyden family. In particular, this subclass includes the well-known DFP, BFGS, and SR1 updates. However, in contrast to the classical quasi-Newton methods, which use the difference of successive iterates for updating the Hessian approximations, our methods apply basis vectors, greedily selected so as to maximize a certain measure of progress. For greedy quasi-Newton methods, we establish an explicit nonasymptotic bound on their rate of local superlinear convergence, as applied to minimizing strongly convex and strongly self-concordant functions (and, in particular, to strongly convex functions with Lipschitz continuous Hessian). The established superlinear convergence rate contains a contraction factor, which depends on the square of the iteration counter. We also show that greedy quasi-Newton methods produce Hessian approximations whose deviation from the exact Hessians linearly converges to zero.

Composite convex optimization with global and local inexact oracles

We introduce new global and local inexact oracle concepts for a wide class of convex functions in composite convex minimization. Such inexact oracles naturally come from primal-dual framework, barrier smoothing, inexact computations of gradients and Hessian, and many other situations. We also provide examples showing that the class of convex functions equipped with the newly inexact second-order oracles is larger than standard self-concordant as well as Lipschitz gradient function classes. Further, we investigate several properties of convex and/or self-concordant functions under the inexact second-order oracles which are useful for algorithm development. Next, we apply our theory to develop inexact proximal Newton-type schemes for minimizing general composite convex minimization problems equipped with such inexact oracles. Our theoretical results consist of new optimization algorithms, accompanied with global convergence guarantees to solve a wide class of composite convex optimization problems. When the first objective term is additionally self-concordant, we establish different local convergence results for our method. In particular, we prove that depending on the choice of accuracy levels of the inexact second-order oracles, we obtain different local convergence rates ranging from $R$-linear and $R$-superlinear to $R$-quadratic. In special cases, where convergence bounds are known, our theory recovers the best known rates. We also apply our settings to derive a new primal-dual method for composite convex minimization problems. Finally, we present some representative numerical examples to illustrate the benefit of our new algorithms.

Worst-Case Convergence Analysis of Inexact Gradient and Newton Methods Through Semidefinite Programming Performance Estimation

We provide new tools for worst-case performance analysis of the gradient (or steepest descent) method of Cauchy for smooth strongly convex functions, and Newton's method for self-concordant functions, including the case of inexact search directions. The analysis uses semidefinite programming performance estimation, as pioneered by Drori and Teboulle [it Math. Program., 145 (2014), pp. 451--482], and extends recent performance estimation results for the method of Cauchy by the authors [it Optim. Lett., 11 (2017), pp. 1185--1199]. To illustrate the applicability of the tools, we demonstrate a novel complexity analysis of short step interior point methods using inexact search directions. As an example in this framework, we sketch how to give a rigorous worst-case complexity analysis of a recent interior point method by Abernethy and Hazan [it PMLR, 48 (2016), pp. 2520--2528].

Quasi-Newton methods: superlinear convergence without line searches for self-concordant functions

ABSTRACTWe consider the use of a curvature-adaptive step size in gradient-based iterative methods, including quasi-Newton methods, for minimizing self-concordant functions, extending an approach first proposed for Newton's method by Nesterov. This step size has a simple expression that can be computed analytically; hence, line searches are not needed. We show that using this step size in the BFGS method (and quasi-Newton methods in the Broyden convex class other than the DFP method) results in superlinear convergence for strongly convex self-concordant functions. We present numerical experiments comparing gradient descent and BFGS methods using the curvature-adaptive step size to traditional methods on deterministic logistic regression problems, and to versions of stochastic gradient descent on stochastic optimization problems.

Generalized self-concordant functions: a recipe for Newton-type methods

We study the smooth structure of convex functions by generalizing a powerful concept so-called self-concordance introduced by Nesterov and Nemirovskii in the early 1990s to a broader class of convex functions which we call generalized self-concordant functions. This notion allows us to develop a unified framework for designing Newton-type methods to solve convex optimization problems. The proposed theory provides a mathematical tool to analyze both local and global convergence of Newton-type methods without imposing unverifiable assumptions as long as the underlying functionals fall into our class of generalized self-concordant functions. First, we introduce the class of generalized self-concordant functions which covers the class of standard self-concordant functions as a special case. Next, we establish several properties and key estimates of this function class which can be used to design numerical methods. Then, we apply this theory to develop several Newton-type methods for solving a class of smooth convex optimization problems involving generalized self-concordant functions. We provide an explicit step-size for a damped-step Newton-type scheme which can guarantee a global convergence without performing any globalization strategy. We also prove a local quadratic convergence of this method and its full-step variant without requiring the Lipschitz continuity of the objective Hessian mapping. Then, we extend our result to develop proximal Newton-type methods for a class of composite convex minimization problems involving generalized self-concordant functions. We also achieve both global and local convergence without additional assumptions. Finally, we verify our theoretical results via several numerical examples, and compare them with existing methods.

Newton Sketch: A Near Linear-Time Optimization Algorithm with Linear-Quadratic Convergence

We propose a randomized second-order method for optimization known as the Newton sketch: it is based on performing an approximate Newton step using a randomly projected Hessian. For self-concordant functions, we prove that the algorithm has superlinear convergence with exponentially high probability, with convergence and complexity guarantees that are independent of condition numbers and related problem-dependent quantities. Given a suitable initialization, similar guarantees also hold for strongly convex and smooth objectives without self-concordance. When implemented using randomized projections based on a subsampled Hadamard basis, the algorithm typically has substantially lower complexity than Newton's method. We also describe extensions of our methods to programs involving convex constraints that are equipped with self-concordant barriers. We discuss and illustrate applications to linear programs, quadratic programs with convex constraints, logistic regression, and other generalized linear models, as...

Inexact proximal Newton methods for self-concordant functions

We analyze the proximal Newton method for minimizing a sum of a self-concordant function and a convex function with an inexpensive proximal operator. We present new results on the global and local convergence of the method when inexact search directions are used. The method is illustrated with an application to L1-regularized covariance selection, in which prior constraints on the sparsity pattern of the inverse covariance matrix are imposed. In the numerical experiments the proximal Newton steps are computed by an accelerated proximal gradient method, and multifrontal algorithms for positive definite matrices with chordal sparsity patterns are used to evaluate gradients and matrix-vector products with the Hessian of the smooth component of the objective.

Contributors

Contributors

Spectral self-concordant functions in the space of two-by-two symmetric matrices

We show that given a two-variable, symmetric, ϑ-self-concordant function f, the spectral function F = f ○ λ is 2(1 + 3ϑ)-self-concordant. Correspondingly, if f is ϑ-self-concordant barrier, then 4(1 + 3ϑ)2 F is a 4(1 + 3ϑ)2ϑ-self-concordant barrier.

Self-concordant analysis for logistic regression

Most of the non-asymptotic theoretical work in regression is carried out for the square loss, where estimators can be obtained through closed-form expressions. In this paper, we use and extend tools from the convex optimization literature, namely self-concordant functions, to provide simple extensions of theoretical results for the square loss to the logistic loss. We apply the extension techniques to logistic regression with regularization by the ℓ2-norm and regularization by the ℓ1-norm, showing that new results for binary classification through logistic regression can be easily derived from corresponding results for least-squares regression.

Polynomial-time interior-point algorithm based on a local self-concordant finite barrier function

The choice of self-concordant functions is the key to efficient algorithms for linear and quadratic convex optimizations, which provide a method with polynomial-time iterations to solve linear and quadratic convex optimization problems. The parameters of a self-concordant barrier function can be used to compute the complexity bound of the proposed algorithm. In this paper, it is proved that the finite barrier function is a local self-concordant barrier function. By deriving the local values of parameters of this barrier function, the desired complexity bound of an interior-point algorithm based on this local self-concordant function for linear optimization problem is obtained. The bound matches the best known bound for small-update methods.

A Globally Convergent Conjugate Gradient Method for Minimizing Self-Concordant Functions On Riemannian Manifolds

Self-concordant functions are a special class of convex functions in Euclidean space introduced by Nesterov. They are used in interior point methods, based on Newton iterations, where they play an important role in solving efficiently certain constrained optimization problems. The concept of self-concordant functions has been defined on Riemannian manifolds by Jiang et al. and a damped Newton method developed for this context. As a further development, this paper proposes a damped conjugate gradient method, which is an ordinary conjugate gradient method but with a novel step-size selection rule which is proved to ensure the algorithm converges to the global minimum. The advantage of the damped conjugate gradient algorithm over the damped Newton method is that the former has a lower computational complexity. To illustrate the advantages, the algorithm is applied to find the center of mass of given points on a hyperboloid model, known as the Karcher mean.