Learning in brain and machine—complexity, Gödel, Aristotle

Abstract

COMBINATORIAL COMPLEXITY OF LEARNING Brains learn much better than computers, this has been discussed in a number of reviews on artificial intelligence, pattern recognitions, and neural networks (Perlovsky, 2001, 2006a). But why? Is there a fundamental reason behind computers being slow learners? Often slow learning is discussed in terms of computational complexity (Perlovsky, 1998), which is usually measured by the number of operations. Scientists have thought that faster computers would be able to catch up with the brain. Still, this has not happened despite computers becoming increasingly faster. Reviews (Perlovsky, 2001, 2006a; Perlovsky et al., 2011) have explained why: computational complexity of learning algorithms grows as a combinatorial (exponential) function of the complexity of a problem to be learned. This means that a learning algorithm might look like it’s quite capable of learning, and indeed, it learns solutions to simple problems. However, slightly more complex problems require not just slightly more computations, but require significantly more. So much more, in fact that learning problems of average complexity require more learning examples and more computer operations than all of the interactions of all elementary particles in the entire life of the Universe (In this article such complexity is called “practically infinite.”). The reason for combinatorial complexity can be explained as follows: consider first, an example of a simple problem requiring no combinatorial complexity for learning: recognition of a single isolated object, which always appears exactly the same. Learning consists in storing in memory the object’s image. Recognition consists in matching the stored image to a newly presented image: match or no match. The complexity of this algorithm approximately equals the number of pixels in an image. But in a real situation the object is not always exactly same; the algorithm has to account for variations in viewing angles, distance, color, etc. In addition, other objects are present with their variabilities. Combinations of various objects with their variabilities lead to combinatorial complexity. Combinations of all pixels in the field of view should be considered. A human eye senses ∼10,000 pixels 10 times a second. Today, sensors measure millions of pixels each second (or more). The number of combinations of these pixels is “practically infinite”; combinations of 100 pixels (a relatively simple problem) are 100100; this number is close to all of the interactions of all elementary particles in the entire life of the Universe.