Distribution-Free Learning

Abstract

In the preceding chapter we studied the learning problem in the case where the random samples were generated by a known fixed probability measure. The problem studied in the present chapter can in some sense be thought of as being at the other end of the spectrum. The focus here is on so-called distribution-free learning; that is, the probability measure generating the samples can be any probability measure on the underlying measurable space. In other words, there is a complete absence of any prior knowledge about the underlying probability measure. In many ways this assumption is somewhat extreme — it is perhaps reasonable to assume at least a little prior knowledge about the probability measure that is generating the learning samples. Nevertheless, as a learning problem the distribution-free case is very “clean” in that it is possible to derive simple necessary and/or sufficient conditions for learnability involving the VC-dimension or the P-dimension. Moreover, we will see in Chapter 8 that, under suitable conditions, a little prior knowledge about the probability does not really help, in the following sense: Learning when there is so-called nonparametric uncertainty about the underlying probability measure is as difficult as distribution-free learning. One could perhaps argue that the positive results in distribution-free learnability are more meaningful than the negative results.KeywordsProbability MeasureLoss FunctionFunction ClassUniform ConvergenceSample ComplexityThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.