A Theory of Learning Simple Concepts Under Simple Distributions

Abstract

We aim at developing a learning theory where ‘simple’ concepts are easily learnable. In Valiant's learning model, many things appear to be hard to learn. Relatively few concept classes were shown to be learnable in polynomial time. In practice, almost nothing we care to learn appears to be not learnable. To model human learning more closely, we impose a reasonable restriction on Valiant model. We assume that learning happens under an arbitrary simple distribution, rather than an arbitrary distribution as assumed by Valiant [V]. A distribution is simple if it is dominated by a semi-computable distribution. Such an assumption appears to be not very restrictive in many practical cases. Most distributions we customarily deal with are computable or can be approximated or dominated by computable or semi-computable ones, hence they fit our assumption. However such an assumption, amazingly, allows us to exhibit a rich mathematical structure on learning. We systematically develop a general theory of learning under simple distributions. In particular we show that one can learn under all simple distributions, if one can learn under one fixed (universal) distribution. We present interesting learning algorithms and several quite general new learnable classes. These classes are more general than the classes known to be polynomial time learnable in Valiant's original model.