A connectionist learning algorithm with provable generalization and scaling bounds

Abstract

A connectionist learning algorithm, the bounded, randomized, distributed (BRD) algorithm, is presented and formally analyzed within the framework of computational learning theory. From a neural network viewpoint this framework gives clear definitions to commonly used terms such as “generalization” and “scaling up,” and addresses the following questions: • • What class of functions is being learned? • • How many training examples should be used? • • How many iterations are required? • • With what certainty can we be assured of learning a good model? From a computational learning theory perspective, a new class of connectionist concepts is shown to be polynomially learnable using the BRD algorithm. Since a variant of the BRD algorithm is in current use for tasks such as pattern recognition, this makes it one of the few learning algorithms shown to be polynomial within the computational learning theory framework that is close to an “industrial strength” algorithm. The algorithm can fail for several reasons: (a) noisy inputs; (b) underestimation of the difficulty of the concept being learned (i.e., larger concept class required); or (c) bad luck. Whenever the algorithm fails, there are several “fallback bounds” available. Finally, the Appendix gives a learnable class of network functions that strictly enlarges a class of learnable functions, Rivest's k-decision lists.