Saddle Property Research Articles

The effect of smooth parametrizations on nonconvex optimization landscapes

AbstractWe develop new tools to study landscapes in nonconvex optimization. Given one optimization problem, we pair it with another by smoothly parametrizing the domain. This is either for practical purposes (e.g., to use smooth optimization algorithms with good guarantees) or for theoretical purposes (e.g., to reveal that the landscape satisfies a strict saddle property). In both cases, the central question is: how do the landscapes of the two problems relate? More precisely: how do desirable points such as local minima and critical points in one problem relate to those in the other problem? A key finding in this paper is that these relations are often determined by the parametrization itself, and are almost entirely independent of the cost function. Accordingly, we introduce a general framework to study parametrizations by their effect on landscapes. The framework enables us to obtain new guarantees for an array of problems, some of which were previously treated on a case-by-case basis in the literature. Applications include: optimizing low-rank matrices and tensors through factorizations; solving semidefinite programs via the Burer–Monteiro approach; training neural networks by optimizing their weights and biases; and quotienting out symmetries.

Linear Regularizers Enforce the Strict Saddle Property

Linear Regularizers Enforce the Strict Saddle Property

Geometric Analysis of Noisy Low-Rank Matrix Recovery in the Exact Parametrized and the Overparametrized Regimes

The matrix sensing problem is an important low-rank optimization problem that has found a wide range of applications, such as matrix completion, phase synchornization/retrieval, robust principal component analysis (PCA), and power system state estimation. In this work, we focus on the general matrix sensing problem with linear measurements that are corrupted by random noise. We investigate the scenario where the search rank r is equal to the true rank [Formula: see text] of the unknown ground truth (the exact parametrized case), as well as the scenario where r is greater than [Formula: see text] (the overparametrized case). We quantify the role of the restricted isometry property (RIP) in shaping the landscape of the nonconvex factorized formulation and assisting with the success of local search algorithms. First, we develop a global guarantee on the maximum distance between an arbitrary local minimizer of the nonconvex problem and the ground truth under the assumption that the RIP constant is smaller than [Formula: see text]. We then present a local guarantee for problems with an arbitrary RIP constant, which states that any local minimizer is either considerably close to the ground truth or far away from it. More importantly, we prove that this noisy, overparametrized problem exhibits the strict saddle property, which leads to the global convergence of perturbed gradient descent algorithm in polynomial time. The results of this work provide a comprehensive understanding of the geometric landscape of the matrix sensing problem in the noisy and overparametrized regime. Funding: This work was supported by grants from the National Science Foundation, Office of Naval Research, Air Force Office of Scientific Research, and Army Research Office.

Sharp Restricted Isometry Property Bounds for Low-Rank Matrix Recovery Problems with Corrupted Measurements

In this paper, we study a general low-rank matrix recovery problem with linear measurements corrupted by some noise. The objective is to understand under what conditions on the restricted isometry property (RIP) of the problem local search methods can find the ground truth with a small error. By analyzing the landscape of the non-convex problem, we first propose a global guarantee on the maximum distance between an arbitrary local minimizer and the ground truth under the assumption that the RIP constant is smaller than 1/2. We show that this distance shrinks to zero as the intensity of the noise reduces. Our new guarantee is sharp in terms of the RIP constant and is much stronger than the existing results. We then present a local guarantee for problems with an arbitrary RIP constant, which states that any local minimizer is either considerably close to the ground truth or far away from it. Next, we prove the strict saddle property, which guarantees the global convergence of the perturbed gradient descent method in polynomial time. The developed results demonstrate how the noise intensity and the RIP constant of the problem affect the landscape of the problem.

Local and Global Linear Convergence of General Low-Rank Matrix Recovery Problems

We study the convergence rate of gradient-based local search methods for solving low-rank matrix recovery problems with general objectives in both symmetric and asymmetric cases, under the assumption of the restricted isometry property. First, we develop a new technique to verify the Polyak-Lojasiewicz inequality in a neighborhood of the global minimizers, which leads to a local linear convergence region for the gradient descent method. Second, based on the local convergence result and a sharp strict saddle property proven in this paper, we present two new conditions that guarantee the global linear convergence of the perturbed gradient descent method. The developed local and global convergence results provide much stronger theoretical guarantees than the existing results. As a by-product, this work significantly improves the existing bounds on the RIP constant required to guarantee the non-existence of spurious solutions.

Proximal Methods Avoid Active Strict Saddles of Weakly Convex Functions

We introduce a geometrically transparent strict saddle property for nonsmooth functions. This property guarantees that simple proximal algorithms on weakly convex problems converge only to local minimizers, when randomly initialized. We argue that the strict saddle property may be a realistic assumption in applications, since it provably holds for generic semi-algebraic optimization problems.

The Global Optimization Geometry of Low-Rank Matrix Optimization

This paper considers general rank-constrained optimization problems that minimize a general objective function <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${f}( {X})$ </tex-math></inline-formula> over the set of rectangular <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${n}\times {m}$ </tex-math></inline-formula> matrices that have rank at most r. To tackle the rank constraint and also to reduce the computational burden, we factorize <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$ {X}$ </tex-math></inline-formula> into <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$ {U} {V} ^{\mathrm {T}}$ </tex-math></inline-formula> where <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$ {U}$ </tex-math></inline-formula> and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$ {V}$ </tex-math></inline-formula> are <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${n}\times {r}$ </tex-math></inline-formula> and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${m}\times {r}$ </tex-math></inline-formula> matrices, respectively, and then optimize over the small matrices <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$ {U}$ </tex-math></inline-formula> and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$ {V}$ </tex-math></inline-formula> . We characterize the global optimization geometry of the nonconvex factored problem and show that the corresponding objective function satisfies the robust strict saddle property as long as the original objective function f satisfies restricted strong convexity and smoothness properties, ensuring global convergence of many local search algorithms (such as noisy gradient descent) in polynomial time for solving the factored problem. We also provide a comprehensive analysis for the optimization geometry of a matrix factorization problem where we aim to find <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${n}\times {r}$ </tex-math></inline-formula> and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${m}\times {r}$ </tex-math></inline-formula> matrices <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$ {U}$ </tex-math></inline-formula> and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$ {V}$ </tex-math></inline-formula> such that <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$ {U} {V} ^{\mathrm {T}}$ </tex-math></inline-formula> approximates a given matrix <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$ {X}^\star $ </tex-math></inline-formula> . Aside from the robust strict saddle property, we show that the objective function of the matrix factorization problem has no spurious local minima and obeys the strict saddle property not only for the exact-parameterization case where <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\mathrm {rank}( {X}^\star) = {r}$ </tex-math></inline-formula> , but also for the over-parameterization case where <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\mathrm {rank}( {X}^\star) &lt; {r}$ </tex-math></inline-formula> and the under-parameterization case where <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\mathrm {rank}( {X}^\star) &gt; {r}$ </tex-math></inline-formula> . These geometric properties imply that a number of iterative optimization algorithms (such as gradient descent) converge to a global solution with random initialization.

The Global Geometry of Centralized and Distributed Low-rank Matrix Recovery Without Regularization

Low-rank matrix recovery is a fundamental problem in signal processing and machine learning. A recent very popular approach to recovering a low-rank matrix X is to factorize it as a product of two smaller matrices, i.e., X = UV^T, and then optimize over U, V instead of X. Despite the resulting non-convexity, recent results have shown that many factorized objective functions actually have benign global geometry---with no spurious local minima and satisfying the so-called strict saddle property---ensuring convergence to a global minimum for many local-search algorithms. Such results hold whenever the original objective function is restricted strongly convex and smooth. However, most of these results actually consider a modified cost function that includes a balancing regularizer. While useful for deriving theory, this balancing regularizer does not appear to be necessary in practice. In this work, we close this theory-practice gap by proving that the unaltered factorized non-convex problem, without the balancing regularizer, also has similar benign global geometry. Moreover, we also extend our theoretical results to the field of distributed optimization.

Global Optimality in Low-Rank Matrix Optimization

This paper considers the minimization of a general objective function $f(X)$ over the set of rectangular $n\times m$ matrices that have rank at most $r$. To reduce the computational burden, we factorize the variable $X$ into a product of two smaller matrices and optimize over these two matrices instead of $X$. Despite the resulting nonconvexity, recent studies in matrix completion and sensing have shown that the factored problem has no spurious local minima and obeys the so-called strict saddle property (the function has a directional negative curvature at all critical points but local minima). We analyze the global geometry for a general and yet well-conditioned objective function $f(X)$ whose restricted strong convexity and restricted strong smoothness constants are comparable. In particular, we show that the reformulated objective function has no spurious local minima and obeys the strict saddle property. These geometric properties imply that a number of iterative optimization algorithms (such as gradient descent) can provably solve the factored problem with global convergence.

The equilibrium stability for a smooth and discontinuous oscillator with dry friction

In this paper, we investigate the equilibrium stability of a Filippov-type system having multiple stick regions based upon a smooth and discontinuous (SD) oscillator with dry friction. The sets of equilibrium states of the system are analyzed together with Coulomb friction conditions in both $$(f_{\mathrm{n}}, f_{\mathrm{s}})$$ and $$(x, {\dot{x}})$$ planes. In the stability analysis, Lyapunov functions are constructed to derive the instability for the equilibrium set of the hyperbolic type and LaSalle’s invariance principle is employed to obtain the stability of the non-hyperbolic type. Analysis demonstrates the existence of a thick stable manifold and the interior stability of the hyperbolic equilibrium set due to the attractive sliding mode of the Filippov property, and also shows that the unstable manifolds of the hyperbolic-type are that of the endpoints with their saddle property. Numerical calculations show a good agreement with the theoretical analysis and an excellent efficiency of the approach for equilibrium states in this particular Filippov system. Furthermore, the equilibrium bifurcations are presented to demonstrate the transition between the smooth and the discontinuous regimes.

Invariant Manifolds and Orbit Control in the Solar Sail Three-Body Problem

In this paper, we consider issues regarding the control and orbit transfer of solar sails in the circular restricted Earth-sun system. Fixed points for solar sails in this system have the linear dynamic properties of saddles crossed with centers; thus, the fixed points are dynamically unstable and control is required. A natural mechanism of control presents itself: variations in the sail's orientation. We describe an optimal controller to control the sail onto fixed points and periodic orbits about fixed points. We find this controller to be very robust, and we define sets of initial data using spherical coordinates to get a sense of the domain of controllability; we also perform a series of tests for control onto periodic orbits. We then present some mission strategies involving transfers from the Earth to fixed points and onto periodic orbits and involving controlled heteroclinic transfers between fixed points on opposite sides of the Earth. Finally, we present some novel methods for finding periodic orbits in circumstances in which traditional methods break down, based on considerations of the center manifold theorem.

Investigation of saddle trajectories for cardiac CT imaging in cone-beam geometry

This paper investigates cone-beam tomography for a wide class of x-ray source trajectories called saddles. In particular, a mathematical analysis of the number of intersections between a saddle and an arbitrary plane is given. This analysis demonstrates that axially truncated cone-beam projections acquired along a saddle can be used for exact reconstruction at any point in a large volume. The reconstruction can be achieved either using a new algorithm presented herein or using a formula recently introduced by Katsevich (2003 Int. J. Math. Math. Sci. 21 1305–21). The shape of the reconstructed volume and the properties of saddles make saddles attractive for cardiac imaging. Three examples of saddles are presented with a discussion of implementation on devices similar to modern C-arm systems and multislice CT scanners. Reconstruction with one of these saddles has been tested using computer-simulated data, with and without truncation. The imaged phantom for the truncated data is a FORBILD head phantom (representing the heart) that has been modified and embedded inside the FORBILD thorax phantom. The non-truncated data were generated by excluding the thorax. The reconstructed images demonstrate the accuracy of the mathematical results.

Saddles and dynamics in a solvable mean-field model

We use the saddle-approach, recently introduced in the numerical investigation of simple model liquids, in the analysis of a mean-field solvable system. The investigated system is the k-trigonometric model, a k-body interaction mean field system, that generalizes the trigonometric model introduced by Madan and Keyes [J. Chem. Phys. 98, 3342 (1993)] and that has been recently introduced to investigate the relationship between thermodynamics and topology of the configuration space. We find a close relationship between the properties of saddles (stationary points of the potential energy surface) visited by the system and the dynamics. In particular the temperature dependence of saddle order follows that of the diffusivity, both having an Arrhenius behavior at low temperature and a similar shape in the whole temperature range. Our results confirm the general usefulness of the saddle-approach in the interpretation of dynamical processes taking place in interacting systems.

Quasisaddles of liquids: Computational study of a bulk Lennard-Jones system

Inherent saddles of the potential energy surface, U, of a liquid are defined as configurations which correspond to the absolute minima of the pseudopotential surface, W=|∇U|2. Given finite numerical precision, multidimensional minimization procedures will sample both absolute and low-lying minima which are referred to collectively as quasisaddles. The sensitivity of statistical properties of these quasisaddles to the convergence criteria of the minimization procedure is investigated using, as a test system, a simple liquid bound by a quadratically shifted Lennard-Jones pair potential. The variation in statistical properties of quasisaddles is studied over a range of error tolerances spanning five orders of magnitude. Based on our results, it is clear that there are no qualitative changes in statistical properties of saddles over this range of error tolerances and even the quantitative changes are small. The results also show that it is not possible to set up an unambiguous numerical criterion to classify the quasisaddles into true saddles which contain no zero curvature, nontranslational normal modes, and inflexion points which have one or more zero-curvature normal mode directions.

Implementing bounds-based approximations in convex-concave two-stage stochastic programming

This paper is concerned with implementational issues and computational testing of bounds-based approximations for solving two-stage stochastic programs with fixed recourse. The implemented bounds are those derived by the authors previously, using first and cross moment information of the random parameters and a convex-concave saddle property of the recourse function. The paper first examines these bounds with regard to their tightness, monotonic behavior, convergence properties, and computationally exploitable decomposition structures. Subsequently, the bounds are implemented under various partitioning/refining strategies for the successive approximation. The detailed numerical experiments demonstrate the effectiveness in solving large scenario-based two-stage stochastic optimization problems throughsuccessive scenario clusters induced by refining the approximations.

Extremum principles for a general class of saddle functionals

The usual notion of a saddle functional in the calculus of variations assumes a vex/concave structure over the product space of two inner product spaces. Here the ideas extended to include some convexity in both spaces whilst still retaining an overall saddle property. Dual extremum principles are established for these functionals. Examples include periodic solutions of Duffing's equation, an iterative scheme and a pair of simultaneous partial differential equations which arise in magnetohydrodynamics.