Bias of Homotopic Gradient Descent for the Hinge Loss

Abstract

Gradient descent is a simple and widely used optimization method for machine learning. For homogeneous linear classifiers applied to separable data, gradient descent has been shown to converge to the maximal-margin (or equivalently, the minimal-norm) solution for various smooth loss functions. The previous theory does not, however, apply to the non-smooth hinge loss which is widely used in practice. Here, we study the convergence of a homotopic variant of gradient descent applied to the hinge loss and provide explicit convergence rates to the maximal-margin solution for linearly separable data.