Abstract
Let X be a Riemannian manifold and xn a sequence of points in X. Assume that we know a priori some properties of the set A of cluster points of xn. The question is under what conditions that xn will converge. An answer to this question serves to understand the convergence behaviour for iterative algorithms for (constrained) optimisation problems, with many applications such as in Deep Learning. We will explore this question, and show by some examples that having X a submanifold (more generally, a metric subspace) of a good Riemannian manifold (even in infinite dimensions) can greatly help.
Highlights
In this paper we will explain how the geometry of submanifolds of Rk is useful to optimisation problems in Deep Learning, and we explore similar properties for other manifolds
We recall in detail the update rule in Gradient Descent methods (GD)
A disadvantage of Standard GD is that it does not guarantee convergence, to have good behaviour one must assume that f is in CL1,1, that is ∇ f is globally Lipschitz continuous with the Lipschitz constant L, and further assume that δ0 is in the order of 1/L
Summary
We will explore this question, and show by some examples that having X a submanifold (more generally, a metric subspace) of a good Riemannian manifold (even in infinite dimensions) can greatly help. In this paper we will explain how the geometry of submanifolds of Rk is useful to optimisation problems in Deep Learning, and we explore similar properties for other manifolds. For deep neural networks to be able to work, one has to solve large scale and non-convex optimisation.
Sign-in/Register to view highlights & summary Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
