Second-order Stationary Points Research Articles

Sample Complexity of Policy Gradient Finding Second-Order Stationary Points

The policy-based reinforcement learning (RL) can be considered as maximization of its objective. However, due to the inherent non-concavity of its objective, the policy gradient method to a first-order stationary point (FOSP) cannot guar- antee a maximal point. A FOSP can be a minimal or even a saddle point, which is undesirable for RL. It has be found that if all the saddle points are strict, all the second-order station- ary points (SOSP) are exactly equivalent to local maxima. Instead of FOSP, we consider SOSP as the convergence criteria to characterize the sample complexity of policy gradient. Our result shows that policy gradient converges to an (ε, √εχ)-SOSP with probability at least 1 − O(δ) after the total cost of O(ε−9/2)sinificantly improves the state of the art cost O(ε−9).Our analysis is based on the key idea that decomposes the parameter space Rp into three non-intersected regions: non-stationary point region, saddle point region, and local optimal region, then making a local improvement of the objective of RL in each region. This technique can be potentially generalized to extensive policy gradient methods. For the complete proof, please refer to https://arxiv.org/pdf/2012.01491.pdf.

On Nonconvex Optimization for Machine Learning

Gradient descent (GD) and stochastic gradient descent (SGD) are the workhorses of large-scale machine learning. While classical theory focused on analyzing the performance of these methods in convex optimization problems, the most notable successes in machine learning have involved nonconvex optimization, and a gap has arisen between theory and practice. Indeed, traditional analyses of GD and SGD show that both algorithms converge to stationary points efficiently. But these analyses do not take into account the possibility of converging to saddle points. More recent theory has shown that GD and SGD can avoid saddle points, but the dependence on dimension in these analyses is polynomial. For modern machine learning, where the dimension can be in the millions, such dependence would be catastrophic. We analyze perturbed versions of GD and SGD and show that they are truly efficient—their dimension dependence is only polylogarithmic. Indeed, these algorithms converge to second-order stationary points in essentially the same time as they take to converge to classical first-order stationary points.

Linearized ADMM Converges to Second-Order Stationary Points for Non-Convex Problems

In this work, a gradient-based primal-dual method of multipliers is proposed for solving a class of linearly constrained non-convex problems. We show that with random initialization of the primal and dual variables, the algorithm is able to compute second-order stationary points (SOSPs) with probability one. Further, we present applications of the proposed method in popular signal processing and machine learning problems such as decentralized matrix factorization and decentralized training of overparameterized neural networks. One of the key steps in the analysis is to construct a new loss function for these problems such that the required convergence conditions (especially the gradient Lipschitz conditions) can be satisfied without changing the global optimal points.

Distributed Learning in Non-Convex Environments—Part I: Agreement at a Linear Rate

Driven by the need to solve increasingly complex optimization problems in signal processing and machine learning, there has been increasing interest in understanding the behavior of gradient-descent algorithms in non-convex environments. Most available works on distributed non-convex optimization problems focus on the deterministic setting where exact gradients are available at each agent. In this work and its Part II, we consider stochastic cost functions, where exact gradients are replaced by stochastic approximations and the resulting gradient noise persistently seeps into the dynamics of the algorithm. We establish that the diffusion learning strategy continues to yield meaningful estimates non-convex scenarios in the sense that the iterates by the individual agents will cluster in a small region around the network centroid. We use this insight to motivate a short-term model for network evolution over a finite-horizon. In Part II of this work, we leverage this model to establish descent of the diffusion strategy through saddle points in O(1/μ) steps, where μ denotes the step-size, and the return of approximately second-order stationary points in a polynomial number of iterations.

Distributed Learning in Non-Convex Environments— Part II: Polynomial Escape From Saddle-Points

The diffusion strategy for distributed learning from streaming data employs local stochastic gradient updates along with exchange of iterates over neighborhoods. In Part I [3] of this work we established that agents cluster around a network centroid and proceeded to study the dynamics of this point. We established expected descent in non-convex environments in the large-gradient regime and introduced a short-term model to examine the dynamics over finite-time horizons. Using this model, we establish in this work that the diffusion strategy is able to escape from strict saddle-points in O(1/μ) iterations, where μ denotes the step-size; it is also able to return approximately second-order stationary points in a polynomial number of iterations. Relative to prior works on the polynomial escape from saddle-points, most of which focus on centralized perturbed or stochastic gradient descent, our approach requires less restrictive conditions on the gradient noise process.

Necessary and Sufficient Second-Order Optimality Conditions on Hadamard Manifolds

This work is intended to lead a study of necessary and sufficient optimality conditions for scalar optimization problems on Hadamard manifolds. In the context of this geometry, we obtain and present new function types characterized by the property of having all their second-order stationary points be global minimums. In order to do so, we extend the concept convexity in Euclidean space to a more general notion of invexity on Hadamard manifolds. This is done employing notions of second-order directional derivatives, second-order pseudoinvexity functions, and the second-order Karush–Kuhn–Tucker-pseudoinvexity problem. Thus, we prove that every second-order stationary point is a global minimum if and only if the problem is either second-order pseudoinvex or second-order KKT-pseudoinvex depending on whether the problem regards unconstrained or constrained scalar optimization, respectively. This result has not been presented in the literature before. Finally, examples of these new characterizations are provided in the context of “Higgs Boson like” potentials, among others.

Finding Second-Order Stationary Points in Constrained Minimization: A Feasible Direction Approach

This paper introduces a method for computing points satisfying the second-order necessary optimality conditions for nonconvex minimization problems subject to a closed and convex constraint set. The method comprises two independent steps corresponding to the first- and second-order conditions. The first-order step is a generic closed map algorithm, which can be chosen from a variety of first-order algorithms, making it adjustable to the given problem. The second-order step can be viewed as a second-order feasible direction step for nonconvex minimization subject to a convex set. We prove that any limit point of the resulting scheme satisfies the second-order necessary optimality condition, and establish the scheme’s convergence rate and complexity, under standard and mild assumptions. Numerical tests illustrate the proposed scheme.

First-Order Methods for Nonconvex Quadratic Minimization

We consider minimization of indefinite quadratics with either trust-region (norm) constraints or cubic regularization. Despite the nonconvexity of these problems we prove that, under mild assumptions, gradient descent converges to their global solutions, and give a non-asymptotic rate of convergence for the cubic variant. We also consider Krylov subspace solutions and establish sharp convergence guarantees to the solutions of both trust-region and cubic-regularized problems. Our rates mirror the behavior of these methods on convex quadratics and eigenvector problems, highlighting their scalability. When we use Krylov subspace solutions to approximate the cubic-regularized Newton step, our results recover the strongest known convergence guarantees to approximate second-order stationary points of general smooth nonconvex functions.

Computation of second-order directional stationary points for group sparse optimization

ABSTRACTWe consider a nonconvex and nonsmooth group sparse optimization problem where the penalty function is the sum of compositions of a folded concave function and the vector norm for each group variable. We show that under some mild conditions a first-order directional stationary point is a strict local minimizer that fulfils the first-order growth condition, and a second-order directional stationary point is a strong local minimizer that fulfils the second-order growth condition. In order to compute second-order directional stationary points, we construct a twice continuously differentiable smoothing problem and show that any accumulation point of the sequence of second-order stationary points of the smoothing problem is a second-order directional stationary point of the original problem. We give numerical examples to illustrate how to compute a second-order directional stationary point by the smoothing method.

Gradient Descent Finds the Cubic-Regularized Nonconvex Newton Step

We consider the minimization of a nonconvex quadratic form regularized by a cubic term, which may exhibit saddle points and a suboptimal local minimum. Nonetheless, we prove that, under mild assumptions, gradient descent approximates the global minimum to within $\varepsilon$ accuracy in $O(\varepsilon^{-1}\log(1/\varepsilon))$ steps for large $\varepsilon$ and $O(\log(1/\varepsilon))$ steps for small $\varepsilon$ (compared to a condition number we define), with at most logarithmic dependence on the problem dimension. When we use gradient descent to approximate the cubic-regularized Newton step, our result implies a rate of convergence to second-order stationary points of general smooth nonconvex functions.

A decoupled first/second-order steps technique for nonconvex nonlinear unconstrained optimization with improved complexity bounds

In order to be provably convergent towards a second-order stationary point, optimization methods applied to nonconvex problems must necessarily exploit both first and second-order information. However, as revealed by recent complexity analyses of some of these methods, the overall effort to reach second-order points is significantly larger when compared to the one of approaching first-order ones. On the other hand, there are other algorithmic schemes, initially designed with first-order convergence in mind, that do not appear to maintain the same first-order performance when modified to take second-order information into account. In this paper, we propose a technique that separately computes first and second-order steps, and that globally converges to second-order stationary points: it consists in better connecting the steps to be taken and the stationarity criteria, potentially guaranteeing larger steps and decreases in the objective. Our approach is shown to lead to an improvement of the corresponding complexity bound with respect to the first-order optimality tolerance, while having a positive impact on the practical behavior. Although the applicability of our ideas is wider, we focus the presentation on trust-region methods with and without derivatives, and motivate in both cases the interest of our strategy.

A second-order optimality condition with first- and second-order complementarity associated with global convergence of algorithms

We develop a new notion of second-order complementarity with respect to the tangent subspace related to second-order necessary optimality conditions by the introduction of so-called tangent multipliers. We prove that around a local minimizer, a second-order stationarity residual can be driven to zero while controlling the growth of Lagrange multipliers and tangent multipliers, which gives a new second-order optimality condition without constraint qualifications stronger than previous ones associated with global convergence of algorithms. We prove that second-order variants of augmented Lagrangian (under an additional smoothness assumption based on the Lojasiewicz inequality) and interior point methods generate sequences satisfying our optimality condition. We present also a companion minimal constraint qualification, weaker than the ones known for second-order methods, that ensures usual global convergence results to a classical second-order stationary point. Finally, our optimality condition naturally suggests a definition of second-order stationarity suitable for the computation of iteration complexity bounds and for the definition of stopping criteria.

Some theoretical limitations of second-order algorithms for smooth constrained optimization

In second-order algorithms, we investigate the relevance of the constant rank of the full set of active constraints in ensuring global convergence to a second-order stationary point under a constraint qualification. We show that second-order stationarity is not expected in the non-constant rank case if the growth of so-called tangent AKKT2 sequences is not controlled. Since no algorithm controls their growth, we argue that there is a theoretical limitation of algorithms in finding second-order stationary points without constant rank assumptions.

Augmented Lagrangians with constrained subproblems and convergence to second-order stationary points

Augmented Lagrangian methods with convergence to second-order stationary points in which any constraint can be penalized or carried out to the subproblems are considered in this work. The resolution of each subproblem can be done by any numerical algorithm able to return approximate second-order stationary points. The developed global convergence theory is stronger than the ones known for current algorithms with convergence to second-order points in the sense that, besides the flexibility introduced by the general lower-level approach, it includes a loose requirement for the resolution of subproblems. The proposed approach relies on a weak constraint qualification, that allows Lagrange multipliers to be unbounded at the solution. It is also shown that second-order resolution of subproblems increases the chances of finding a feasible point, in the sense that limit points are second-order stationary for the problem of minimizing the squared infeasibility. The applicability of the proposed method is illustrated in numerical examples with ball-constrained subproblems.

A second-order sequential optimality condition associated to the convergence of optimization algorithms

Sequential optimality conditions have recently played an important role on the analysis of the global convergence of optimization algorithms towards first-order stationary points, justifying their stopping criteria. In this paper we introduce a sequential optimality condition that takes into account second-order information and that allows us to improve the global convergence assumptions of several second-order algorithms, which is our main goal. We also present a companion constraint qualification that is less stringent than previous assumptions associated to the convergence of second-order methods, like the joint condition Mangasarian-Fromovitz and Weak Constant Rank. Our condition is also weaker than the Constant Rank Constraint Qualification, which associated it to the convergence of second-order algorithms. This means that we can prove second-order global convergence of well stablished algorithms even when the set of Lagrange multipliers is unbounded, which overcomes a limitation of previous results based on MFCQ. We prove global convergence of well known variations of the augmented Lagrangian and Regularized SQP methods to second-order stationary points under this new weak constraint qualification.

A trust region algorithm with a worst-case iteration complexity of $$\mathcal{O}(\epsilon ^{-3/2})$$ O ( ϵ - 3 / 2 ) for nonconvex optimization

We propose a trust region algorithm for solving nonconvex smooth optimization problems. For any $$\overline{\epsilon }\in (0,\infty )$$∈¯?(0,?), the algorithm requires at most $$\mathcal{O}(\epsilon ^{-3/2})$$O(∈-3/2) iterations, function evaluations, and derivative evaluations to drive the norm of the gradient of the objective function below any $$\epsilon \in (0,\overline{\epsilon }]$$∈?(0,∈¯]. This improves upon the $$\mathcal{O}(\epsilon ^{-2})$$O(∈-2) bound known to hold for some other trust region algorithms and matches the $$\mathcal{O}(\epsilon ^{-3/2})$$O(∈-3/2) bound for the recently proposed Adaptive Regularisation framework using Cubics, also known as the arc algorithm. Our algorithm, entitled trace, follows a trust region framework, but employs modified step acceptance criteria and a novel trust region update mechanism that allow the algorithm to achieve such a worst-case global complexity bound. Importantly, we prove that our algorithm also attains global and fast local convergence guarantees under similar assumptions as for other trust region algorithms. We also prove a worst-case upper bound on the number of iterations, function evaluations, and derivative evaluations that the algorithm requires to obtain an approximate second-order stationary point.

Two globally convergent nonmonotone trust-region methods for unconstrained optimization

This paper addresses some trust-region methods equipped with nonmonotone strategies for solving nonlinear unconstrained optimization problems. More specifically, the importance of using nonmonotone techniques in nonlinear optimization is motivated, then two new nonmonotone terms are proposed, and their combinations into the traditional trust-region framework are studied. The global convergence to first- and second-order stationary points and local superlinear and quadratic convergence rates for both algorithms are established. Numerical experiments on the CUTEst test collection of unconstrained problems and some highly nonlinear test functions are reported, where a comparison among state-of-the-art nonmonotone trust-region methods shows the efficiency of the proposed nonmonotone schemes.

Iterative reweighted minimization methods for $$l_p$$ l p regularized unconstrained nonlinear programming

In this paper we study general $$l_p$$ l p regularized unconstrained minimization problems. In particular, we derive lower bounds for nonzero entries of the first- and second-order stationary points and hence also of local minimizers of the $$l_p$$ l p minimization problems. We extend some existing iterative reweighted $$l_1$$ l 1 ( $$\mathrm{IRL}_1$$ IRL 1 ) and $$l_2$$ l 2 ( $$\mathrm{IRL}_2$$ IRL 2 ) minimization methods to solve these problems and propose new variants for them in which each subproblem has a closed-form solution. Also, we provide a unified convergence analysis for these methods. In addition, we propose a novel Lipschitz continuous $${\epsilon }$$ ∈ -approximation to $$\Vert x\Vert ^p_p$$ ? x ? p p . Using this result, we develop new $$\mathrm{IRL}_1$$ IRL 1 methods for the $$l_p$$ l p minimization problems and show that any accumulation point of the sequence generated by these methods is a first-order stationary point, provided that the approximation parameter $${\epsilon }$$ ∈ is below a computable threshold value. This is a remarkable result since all existing iterative reweighted minimization methods require that $${\epsilon }$$ ∈ be dynamically updated and approach zero. Our computational results demonstrate that the new $$\mathrm{IRL}_1$$ IRL 1 method and the new variants generally outperform the existing $$\mathrm{IRL}_1$$ IRL 1 methods (Chen and Zhou in 2012; Foucart and Lai in Appl Comput Harmon Anal 26:395---407, 2009).

A subclass of generating set search with convergence to second-order stationary points

In generalized pattern search algorithms for linearly constrained optimization, certain limit points have been shown to satisfy a pseudo-second-order necessary condition, in which positive semidefiniteness of the Hessian is achieved only with respect to an orthonormal basis. Satisfaction of the full second-order condition can only be ensured if the finite set of positive spanning directions includes the eigenvectors of the Hessian at the limit point. Of course, this is generally not possible in practice, due to the unavailability of derivatives. The present work takes advantage of a chief feature of the more general parent class of generating set search algorithms – that of allowing the use of an infinite (non-dense) set of directions in the limit. If approximate second derivative information is gathered during the iteration process, then the normalized eigenvectors of the approximate Hessian can be used to construct the set of 2n orthonormal directions used by the algorithm. If the Hessian approximation converges in the limit to the true Hessian (or a matrix in the generalized Hessian), then the full second-order optimality conditions can be achieved. Two approaches for generating such a convergent approximate Hessian are described. Numerical examples show the effectiveness of the approach.

Second-order invex functions in nonlinear programming

We introduce a notion of a second-order invex function. A Fréchet differentiable invex function without any further assumptions is second-order invex. It is shown that the inverse claim does not hold. A Fréchet differentiable function is second-order invex if and only if each second-order stationary point is a global minimizer. Two complete characterizations of these functions are derived. It is proved that a quasiconvex function is second-order invex if and only if it is second-order pseudoconvex. Further, we study the nonlinear programming problem with inequality constraints whose objective function is second-order invex. We introduce a notion of second-order type I objective and constraint functions. This class of problems strictly includes the type I invex ones. Then we extend a lot of sufficient optimality conditions with generalized convex functions to problems with second-order type I invex objective function and constraints. Additional optimality results, which concern type I and second-order type I invex data are obtained. An answer to the question when a kernel, which is not identically equal to zero, exists is given.