Learning with restricted focus of attention

Abstract

We consider learning tasks in which the learner faces restrictions on the amount of information he can extract from each example he encounters. We introduce a formal framework for the analysis of such scenarios. We call this framework RFA (restricted focus of attention) learning. Although it is a natural refinement of the PAC learning model, some of the fundamental PAC-learning results and techniques fail in the RFA paradigm; learnability in the RFA model is no longer characterized by the VC dimension, and many PAC learning algorithms are not applicable in the RFA setting. Hence, the RFA formulation reflects the need for new techniques and tools to cope with some fundamental constraints of realistic learning problems. In this work we also present some paradigms and algorithms that may serve as a first step toward answering this need. Two main types of restrictions are considered here: In the more stringent one, calledk-RFA, onlykof thenattributes of each example are revealed to the learner, while in the more permissive one, calledk-wRFA, the restriction is made on the size of each observation (kbits), and no restriction is made on how the observations are extracted from the examples. For thek-RFA restriction we develop a general technique for composing efficientk-RFA algorithms and apply it to deduce, for instance, the efficientk-RFA learnability ofk-DNF formulas and the efficient 1-RFA learnability of axis-aligned rectangles in the Euclidean spaceRn. We also prove thek-RFA learnability of richer classes of Boolean functions (such ask-decision lists) with respect to a given distribution and the efficient (n?1)-RFA learnability (for fixedn), under product distributions, of classes of subsets ofRnwhich are defined by mild surfaces. For thek-wRFA restriction, we show that fork=O(logn), efficientk-wRFA learning is robust against classification noise. As a straight- forward application, we obtain a new simple noise-tolerant algorithm for the class ofk-decision lists by constructing an intuitivek-wRFA algorithm for this task.