Strict Saddle Points Research Articles

Gradient Descent Provably Escapes Saddle Points in the Training of Shallow ReLU Networks

AbstractDynamical systems theory has recently been applied in optimization to prove that gradient descent algorithms bypass so-called strict saddle points of the loss function. However, in many modern machine learning applications, the required regularity conditions are not satisfied. In this paper, we prove a variant of the relevant dynamical systems result, a center-stable manifold theorem, in which we relax some of the regularity requirements. We explore its relevance for various machine learning tasks, with a particular focus on shallow rectified linear unit (ReLU) and leaky ReLU networks with scalar input. Building on a detailed examination of critical points of the square integral loss function for shallow ReLU and leaky ReLU networks relative to an affine target function, we show that gradient descent circumvents most saddle points. Furthermore, we prove convergence to global minima under favourable initialization conditions, quantified by an explicit threshold on the limiting loss.

On the Global Convergence of Randomized Coordinate Gradient Descent for Nonconvex Optimization

In this work, we analyze the global convergence property of a coordinate gradient descent with random choice of coordinates and stepsizes for nonconvex optimization problems. Under generic assumptions, we prove that the algorithm iterate will almost surely escape strict saddle points of the objective function. As a result, the algorithm is guaranteed to converge to local minima if all saddle points are strict. Our proof is based on viewing the coordinate descent algorithm as a nonlinear random dynamical system and a quantitative finite block analysis of its linearization around saddle points.

From the simplex to the sphere: faster constrained optimization using the Hadamard parametrization

Abstract The standard simplex in $\mathbb{R}^{n}$, also known as the probability simplex, is the set of nonnegative vectors whose entries sum up to 1. It frequently appears as a constraint in optimization problems that arise in machine learning, statistics, data science, operations research and beyond. We convert the standard simplex to the unit sphere and thus transform the corresponding constrained optimization problem into an optimization problem on a simple, smooth manifold. We show that Karush-Kuhn-Tucker points and strict-saddle points of the minimization problem on the standard simplex all correspond to those of the transformed problem, and vice versa. So, solving one problem is equivalent to solving the other problem. Then, we propose several simple, efficient and projection-free algorithms using the manifold structure. The equivalence and the proposed algorithm can be extended to optimization problems with unit simplex, weighted probability simplex or $\ell _{1}$-norm sphere constraints. Numerical experiments between the new algorithms and existing ones show the advantages of the new approach. Open source code is available at https://github.com/DanielMckenzie/HadRGD.

Boundary Conditions for Linear Exit Time Gradient Trajectories Around Saddle Points: Analysis and Algorithm

Gradient-related first-order methods have become the workhorse of large-scale numerical optimization problems. Many of these problems involve nonconvex objective functions with multiple saddle points, which necessitates an understanding of the behavior of discrete trajectories of first-order methods within the geometrical landscape of these functions. This paper concerns convergence of first-order discrete methods to a local minimum of nonconvex optimization problems that comprise strict-saddle points within the geometrical landscape. To this end, it focuses on analysis of discrete gradient trajectories around saddle neighborhoods, derives sufficient conditions under which these trajectories can escape strict-saddle neighborhoods in linear time, explores the contractive and expansive dynamics of these trajectories in neighborhoods of strict-saddle points that are characterized by gradients of moderate magnitude, characterizes the non-curving nature of these trajectories, and highlights the inability of these trajectories to re-enter the neighborhoods around strict-saddle points after exiting them. Based on these insights and analyses, the paper then proposes a simple variant of the vanilla gradient descent algorithm, termed Curvature Conditioned Regularized Gradient Descent (CCRGD) algorithm, which utilizes a check for an initial boundary condition to ensure its trajectories can escape strict-saddle neighborhoods in linear time. Convergence analysis of the CCRGD algorithm, which includes its rate of convergence to a local minimum, is also presented in the paper. Numerical experiments are then provided on a test function as well as a low-rank matrix factorization problem to evaluate the efficacy of the proposed algorithm.

Distributed Synchronous and Asynchronous Algorithms for Semidefinite Programming With Diagonal Constraints

This article develops distributed synchronous and asynchronous algorithms for the large-scale semidefinite programming with diagonal constraints, which has wide applications in combinatorial optimization, image processing, and community detection. The information of the semidefinite programming is allocated to multiple interconnected agents such that each agent aims to find a solution by communicating to its neighbors. Based on the low-rank property of solutions and the Burer–Monteiro factorization, we transform the original problem into a distributed optimization problem over unit spheres to reduce variable dimensions and ensure positive semidefiniteness without involving semidefinite projections, which are computationally expensive. For the distributed optimization problem, we propose distributed synchronous and asynchronous algorithms, both of which reduce computational burden and storage space compared with existing centralized algorithms. Specifically, the distributed synchronous algorithm almost surely escapes strict saddle points and converges to the set of optimal solutions to the optimization problem. In addition, the proposed distributed asynchronous algorithm allows communication delays and converges to critical points to the optimization problem under mild conditions. By applying the proposed algorithms to image segmentation, we illustrate the efficiency and convergence performance of the two proposed algorithms.

Escaping Strict Saddle Points of the Moreau Envelope in Nonsmooth Optimization

Recent work has shown that stochastically perturbed gradient methods can efficiently escape strict saddle points of smooth functions. We extend this body of work to nonsmooth optimization, by analyzing an inexact analogue of a stochastically perturbed gradient method applied to the Moreau envelope. The main conclusion is that a variety of algorithms for nonsmooth optimization can escape strict saddle points of the Moreau envelope at a controlled rate. The main technical insight is that many algorithms applied to the proximal subproblem yield directions that approximate the gradient of the Moreau envelope.

Embedding Principle: A Hierarchical Structure of Loss Landscape of Deep Neural Networks

We prove a general Embedding Principle of loss landscape of deep neural networks (NNs) that unravels a hierarchical structure of the loss landscape of NNs, i.e., loss landscape of an NN contains all critical points of all the narrower NNs. This result is obtained by constructing a class of critical embeddings which map any critical point of a narrower NN to a critical point of the target NN with the same output function. By discovering a wide class of general compatible critical embeddings, we provide a gross estimate of the dimension of critical submanifolds embedded from critical points of narrower NNs. We further prove an irreversiblility property of any critical embedding that the number of negative/zero/positive eigenvalues of the Hessian matrix of a critical point may increase but never decrease as an NN becomes wider through the embedding. Using a special realization of general compatible critical embedding, we prove a stringent necessary condition for being a "truly-bad" critical point that never becomes a strict-saddle point through any critical embedding. This result implies the commonplace of strict-saddle points in wide NNs, which may be an important reason underlying the easy optimization of wide NNs widely observed in practice.

Likelihood landscape and maximum likelihood estimation for the discrete orbit recovery model

AbstractWe study the nonconvex optimization landscape for maximum likelihood estimation in the discrete orbit recovery model with Gaussian noise. This is a statistical model motivated by applications in molecular microscopy and image processing, where each measurement of an unknown object is subject to an independent random rotation from a known rotational group. Equivalently, it is a Gaussian mixture model where the mixture centers belong to a group orbit.We show that fundamental properties of the likelihood landscape depend on the signal‐to‐noise ratio and the group structure. At low noise, this landscape is “benign” for any discrete group, possessing no spurious local optima and only strict saddle points. At high noise, this landscape may develop spurious local optima, depending on the specific group. We discuss several positive and negative examples, and provide a general condition that ensures a globally benign landscape at high noise. For cyclic permutations of coordinates on (multireference alignment), there may be spurious local optima when , and we establish a correspondence between these local optima and those of a surrogate function of the phase variables in the Fourier domain.We show that the Fisher information matrix transitions from resembling that of a single Gaussian distribution in low noise to having a graded eigenvalue structure in high noise, which is determined by the graded algebra of invariant polynomials under the group action. In a local neighborhood of the true object, where the neighborhood size is independent of the signal‐to‐noise ratio, the landscape is strongly convex in a reparametrized system of variables given by a transcendence basis of this polynomial algebra. We discuss implications for optimization algorithms, including slow convergence of expectation‐maximization, and possible advantages of momentum‐based acceleration and variable reparametrization for first‐ and second‐order descent methods. © 2021 Wiley Periodicals LLC.

A Diffusion Approximation Theory of Momentum Stochastic Gradient Descent in Nonconvex Optimization

Momentum stochastic gradient descent (MSGD) algorithm has been widely applied to many nonconvex optimization problems in machine learning (e.g., training deep neural networks, variational Bayesian inference, etc.). Despite its empirical success, there is still a lack of theoretical understanding of convergence properties of MSGD. To fill this gap, we propose to analyze the algorithmic behavior of MSGD by diffusion approximations for nonconvex optimization problems with strict saddle points and isolated local optima. Our study shows that the momentum helps escape from saddle points but hurts the convergence within the neighborhood of optima (if without the step size annealing or momentum annealing). Our theoretical discovery partially corroborates the empirical success of MSGD in training deep neural networks.

Distributed Learning in Non-Convex Environments— Part II: Polynomial Escape From Saddle-Points

The diffusion strategy for distributed learning from streaming data employs local stochastic gradient updates along with exchange of iterates over neighborhoods. In Part I [3] of this work we established that agents cluster around a network centroid and proceeded to study the dynamics of this point. We established expected descent in non-convex environments in the large-gradient regime and introduced a short-term model to examine the dynamics over finite-time horizons. Using this model, we establish in this work that the diffusion strategy is able to escape from strict saddle-points in O(1/μ) iterations, where μ denotes the step-size; it is also able to return approximately second-order stationary points in a polynomial number of iterations. Relative to prior works on the polynomial escape from saddle-points, most of which focus on centralized perturbed or stochastic gradient descent, our approach requires less restrictive conditions on the gradient noise process.

Second-Order Guarantees of Distributed Gradient Algorithms

We consider distributed smooth nonconvex unconstrained optimization over net- works, modeled as a connected graph. We examine the behavior of distributed gradient-based algorithms near strict saddle points. Specifically, we establish that (i) the renowned distributed gradient descent algorithm likely converges to a neighborhood of a second-order stationary (SoS) solution; and (ii) the more recent class of distributed algorithms based on gradient tracking---implementable also over digraphs---likely converges to exact SoS solutions, thus avoiding (strict) saddle points. Furthermore, new convergence rate results for first-order critical points is established for the latter class of algorithms.

Extending the Step-Size Restriction for Gradient Descent to Avoid Strict Saddle Points

We provide larger step-size restrictions for which gradient descent-based algorithms (almost surely) avoid strict saddle points. In particular, consider a twice differentiable (nonconvex) objective...

Combining Stochastic Adaptive Cubic Regularization with Negative Curvature for Nonconvex Optimization

We focus on minimizing nonconvex finite-sum functions that typically arise in machine learning problems. In an attempt to solve this problem, the adaptive cubic-regularized Newton method has shown its strong global convergence guarantees and the ability to escape from strict saddle points. In this paper, we expand this algorithm to incorporating the negative curvature method to update even at unsuccessful iterations. We call this new method Stochastic Adaptive cubic regularization with Negative Curvature (SANC). Unlike the previous method, in order to attain stochastic gradient and Hessian estimators, the SANC algorithm uses independent sets of data points of consistent size over all iterations. It makes the SANC algorithm more practical to apply for solving large-scale machine learning problems. To the best of our knowledge, this is the first approach that combines the negative curvature method with the adaptive cubic-regularized Newton method. Finally, we provide experimental results, including neural networks problems supporting the efficiency of our method.

Structured Local Optima in Sparse Blind Deconvolution

Blind deconvolution is a ubiquitous problem aiming to recover a convolution kernel $\boldsymbol a_{0}\in \mathbb R ^{k}$ and an activation signal $\boldsymbol x_{0}\in \mathbb R ^{m}$ from their convolution $\boldsymbol y\in \mathbb R ^{m}$ . Unfortunately, this is an ill-posed problem in general. This paper focuses on the short and sparse blind deconvolution problem, where the convolution kernel is short ( $k\ll m$ ) and the activation signal is sparsely and randomly supported ( $\left \|{ \boldsymbol x_{0} }\right \|_{0}\ll m$ ). This variant captures the structure of the convolutional signals in several important application scenarios. In this paper, we normalize the convolution kernel to have unit Frobenius norm and then cast the blind deconvolution problem as a nonconvex optimization problem over the kernel sphere. We demonstrate that (i) in a certain region of the sphere, every local optimum is close to some shift truncation of the ground truth, and (ii) for a generic unit kernel $\boldsymbol a_{0}$ , when the sparsity of activation signal satisfies $\theta \lesssim k^{-2/3}$ and number of measurements $m\gtrsim \mathop {\mathrm {poly}}\nolimits \left ({k }\right) $ , the proposed initialization method together with a descent algorithm which escapes strict saddle points recovers some shift truncation of the ground truth kernel.

Symmetry, Saddle Points, and Global Optimization Landscape of Nonconvex Matrix Factorization

We propose a general theory for studying the landscape of nonconvex optimization with underlying symmetric structures for a class of machine learning problems (e.g., low-rank matrix factorization, phase retrieval, and deep linear neural networks). In particular, we characterize the locations of stationary points and the null space of Hessian matrices of the objective function via the lens of invariant groups. As a major motivating example, we apply the proposed general theory to characterize the global landscape of the nonconvex optimization in low-rank matrix factorization problem. We illustrate how the rotational symmetry group gives rise to infinitely many nonisolated strict saddle points and equivalent global minima of the objective function. By explicitly identifying all stationary points, we divide the entire parameter space into three regions: ( $ \mathcal {R}_{1}$ ) the region containing the neighborhoods of all strict saddle points where the objective has negative curvature; ( $ \mathcal {R}_{2}$ ) the region containing neighborhoods of all global minima, where the objective enjoys strong convexity along certain directions; and ( $ \mathcal {R}_{3}$ ) the complement of the above regions, where the gradient has sufficiently large magnitude. We further extend our result to the matrix sensing problem. Such global landscape implies that strong global convergence guarantees for popular iterative algorithms with arbitrary initial solutions.

First-order methods almost always avoid strict saddle points

We establish that first-order methods avoid strict saddle points for almost all initializations. Our results apply to a wide variety of first-order methods, including (manifold) gradient descent, b...

An Envelope for Davis–Yin Splitting and Strict Saddle-Point Avoidance

It is known that operator splitting methods based on forward–backward splitting, Douglas–Rachford splitting, and Davis–Yin splitting decompose difficult optimization problems into simpler subproblems under proper convexity and smoothness assumptions. In this paper, we identify an envelope (an objective function), whose gradient descent iteration under a variable metric coincides with Davis–Yin splitting iteration. This result generalizes the Moreau envelope for proximal-point iteration and the envelopes for forward–backward splitting and Douglas–Rachford splitting iterations identified by Patrinos, Stella, and Themelis. Based on the new envelope and the stable–center manifold theorem, we further show that, when forward–backward splitting or Douglas–Rachford splitting iterations start from random points, they avoid all strict saddle points with probability one. This result extends the similar results by Lee et al. from gradient descent to splitting methods.

Behavior of accelerated gradient methods near critical points of nonconvex functions

We examine the behavior of accelerated gradient methods in smooth nonconvex unconstrained optimization, focusing in particular on their behavior near strict saddle points. Accelerated methods are iterative methods that typically step along a direction that is a linear combination of the previous step and the gradient of the function evaluated at a point at or near the current iterate. (The previous step encodes gradient information from earlier stages in the iterative process). We show by means of the stable manifold theorem that the heavy-ball method is unlikely to converge to strict saddle points, which are points at which the gradient of the objective is zero but the Hessian has at least one negative eigenvalue. We then examine the behavior of the heavy-ball method and other accelerated gradient methods in the vicinity of a strict saddle point of a nonconvex quadratic function, showing that both methods can diverge from this point more rapidly than the steepest-descent method.

Homogenization and non-homogenization of certain non-convex Hamilton–Jacobi equations

We continue the study of the homogenization of coercive non-convex Hamilton–Jacobi equations in random media identifying two general classes of Hamiltonians with very distinct behavior. For the first class there is no homogenization in a particular environment while for the second homogenization takes place in environments with finite range dependence. Motivated by the recent counter-example of Ziliotto, who constructed a coercive but non-convex Hamilton–Jacobi equation with stationary ergodic random potential field for which homogenization does not hold, we show that same happens for coercive Hamiltonians which have a strict saddle-point, a very local property. We also identify, based on the recent work of Armstrong and Cardaliaguet on the homogenization of positively homogeneous random Hamiltonians in environments with finite range dependence, a new general class Hamiltonians, namely equations with uniformly strictly star-shaped sub-level sets, which homogenize.

Strict Efficiency in Vector Optimization with Nearly Convexlike Set-Valued Maps

The concept of the well posedness for a special scalar problem is linked with strictly efficient solutions of vector optimization problem involving nearly convexlike set-valued maps. Two scalarization theorems and two Lagrange multiplier theorems for strict efficiency in vector optimization involving nearly convexlike set-valued maps are established. A dual is proposed and duality results are obtained in terms of strictly efficient solutions. A new type of saddle point, called strict saddle point, of an appropriate set-valued Lagrange map is introduced and is used to characterize strict efficiency.