A note on margin-based loss functions in classification

Abstract

In many classification procedures, the classification function is obtained by minimizing a certain empirical risk on the training sample. The classification is then based on the sign of the classification function. In recent years, there have been a host of classification methods proposed that use different margin-based loss functions. The margin-based loss functions are often motivated as upper bounds of the misclassification loss, but this cannot explain the statistical properties of the classification procedures. We show that a large family of margin-based loss functions are Fisher consistent for classification. That is, the population minimizer of the loss function leads to the Bayes optimal rule of classification. Our result covers almost all margin-based loss functions that have been proposed in the literature. We give an inequality that links the Fisher consistency of margin-based loss functions with the consistency of methods based on these loss functions. We use this inequality to obtain the rate of convergence for the method of sieves based on a class of margin-based loss functions.