Optimal Exponential Bounds on the Accuracy of Classification

Abstract

Consider a standard binary classification problem, in which $$(X,Y)$$ is a random couple in $$\mathcal{X}\times \{0,1\}$$ , and the training data consist of $$n$$ i.i.d. copies of $$(X,Y).$$ Given a binary classifier $$f:\mathcal{X}\mapsto \{0,1\},$$ the generalization error of $$f$$ is defined by $$R(f)={\mathbb P}\{Y\ne f(X)\}$$ . Its minimum $$R^*$$ over all binary classifiers $$f$$ is called the Bayes risk and is attained at a Bayes classifier. The performance of any binary classifier $$\hat{f}_n$$ based on the training data is characterized by the excess risk $$R(\hat{f}_n)-R^*$$ . We study Bahadur’s type exponential bounds on the following minimax accuracy confidence function based on the excess risk: where the supremum is taken over all distributions $$P$$ of $$(X,Y)$$ from a given class of distributions $$\mathcal{M}$$ and the infimum is over all binary classifiers $$\hat{f}_n$$ based on the training data. We study how this quantity depends on the complexity of the class of distributions $$\mathcal{M}$$ characterized by exponents of entropies of the class of regression functions or of the class of Bayes classifiers corresponding to the distributions from $$\mathcal{M}.$$ We also study its dependence on margin parameters of the classification problem. In particular, we show that, in the case when $$\mathcal{X}=[0,1]^d$$ and $$\mathcal{M}$$ is the class of all distributions satisfying the margin condition with exponent $$\alpha >0$$ and such that the regression function $$\eta $$ belongs to a given Holder class of smoothness $$\beta >0,$$ for some constants $$D,\lambda _0>0$$ .