Constrained Stochastic Gradient Descent: The Good Practice

Abstract

Stochastic Gradient Descent (SGD) is the method of choice for large scale problems, most notably in deep learning. Recent studies target improving convergence and speed of the SGD algorithm. In this paper, we equip the SGD algorithm and its advanced versions with an intriguing feature, namely handling constrained problems. Constraints such as orthogonality are pervasive in learning theory. Nevertheless and to some extent surprising, constrained SGD algorithms are rarely studied. Our proposal makes use of Riemannian geometry and accelerated optimization techniques to deliver efficient and constrained-aware SGD methods.We will assess and contrast our proposed approaches in a wide range of problems including incremental dimensionality reduction, karcher mean and deep metric learning.