Improved Hoeffding-style performance guarantees for accurate classifiers

Abstract

The Hoeffding bounds on classifier accuracy depend on the accuracy itself as a parameter. Since the accuracy is unknown, we must use the parameter value that makes them weakest (50% accuracy) to be certain that the bounds are valid. For accurate classifiers, we show that the following process can give stronger bounds. First use the Hoeffding method to derive a loose bound. Then restrict the parameter value to the range of the loose bound to produce a stronger bound.