Learning theory of minimum error entropy under weak moment conditions

Abstract

Minimum error entropy (MEE) is an information theoretic learning approach that minimizes the information contained in the prediction error, which is measured by entropy. It has been successfully used in various machine learning tasks for its robustness to heavy-tailed distributions and outliers. In this paper, we consider its use in nonparametric regression and analyze its generalization performance from a learning theory perspective by imposing a [Formula: see text]th order moment condition on the noise variable. To this end, we establish a comparison theorem to characterize the relation between the excess generalization error and the prediction error. A relaxed Bernstein condition and concentration inequalities are used to derive error bounds and learning rates. Note that the [Formula: see text]th moment condition is rather weak particularly when [Formula: see text] because the noise variable does not even admit a finite variance in this case. Therefore, our analysis explains the robustness of MEE in the presence of heavy-tailed distributions.