Convergence of Gradient Descent for Minimum Error Entropy Principle in Linear Regression

Abstract

We study the convergence of minimum error entropy (MEE) algorithms when they are implemented by gradient descent. This method has been used in practical applications for more than one decade, but there has been no consistency or rigorous error analysis. This paper gives the first rigorous proof for the convergence of the gradient descent method for MEE in a linear regression setting. The mean square error is proved to decay exponentially fast in terms of the iteration steps and of order $O(\frac{1}{m})$ in terms of the sample size $m$ . The mean square convergence is guaranteed when the step size is chosen appropriately and the scaling parameter is large enough.