Computational Learning Theory Research Articles

Convergence of exponentiated gradient algorithms

This paper studies three related algorithms: the (traditional) gradient descent (GD) algorithm, the exponentiated gradient algorithm with positive and negative weights (EG/spl plusmn/ algorithm), and the exponentiated gradient algorithm with unnormalized positive and negative weights (EGU/spl plusmn/ algorithm). These algorithms have been previously analyzed using the framework in the computational learning theory community. We perform a traditional signal processing analysis in terms of the mean square error. A relationship between the learning rate and the mean squared error (MSE) of predictions is found for the family of algorithms. This is used to compare the performance of the algorithms by choosing learning rates such that they converge to the same steady-state MSE. We demonstrate that if the target weight vector is sparse, the EG/spl plusmn/ algorithm typically converges more quickly than the GD or EGU/spl plusmn/ algorithms that perform very similarly. A side effect of our analysis is a reparametrization of the algorithms that provides insights into their behavior. The general form of the results we obtain are consistent with those obtained in the mistake-bound framework. The application of the algorithms to acoustic echo cancellation is then studied, and it is shown in some circumstances that the EG/spl plusmn/ algorithm will converge faster than the other two algorithms.

Learning logic programs with structured background knowledge

The efficient learnability of restricted classes of logic programs is studied in the PAC framework of computational learning theory. We develop the product homomorphism method, which gives polynomial PAC learning algorithms for a nonrecursive Horn clause with function-free ground background knowledge, if the background knowledge satisfies some structural properties. The method is based on a characterization of the concept that corresponds to the relative least general generalization of a set of positive examples with respect to the background knowledge. The characterization is formulated in terms of products and homomorphisms. In the applications this characterization is turned into an explicit combinatorial description, which is then translated into the language of nonrecursive Horn clauses. We show that a nonrecursive Horn clause is polynomially PAC-learnable if there is a single binary background predicate and the ground atoms in the background knowledge form a forest. If the ground atoms in the background knowledge form a disjoint union of cycles then the situation is different, as the shortest consistent hypothesis may have exponential size. In this case polynomial PAC-learnability holds if a different representation language is used. We also consider the complexity of hypothesis finding for multiple clauses in some restricted cases.

A Bayesian boosting theorem

We refine the first theorem of (R.E. Schapire, Y. Singer, in: Proceedings of the 11th Annual ACM Conference on Computational Learning Theory, 1998, pp. 80–91) bounding the error of the A daB oost boosting algorithm, to integrate Bayes risk. This suggests the significant time savings could be obtained on some domains without damaging the solution. An applicative example is given in the field of feature selection.

The Lob–Pass Problem

We consider a new variant of the online learning model in which the goal of an agent is to choose his or her actions so as to maximize the number of successes, while learning about his or her reacting environment through those very actions. In particular, we consider a model of tennis play, in which the only actions that the player can take are a pass and a lob, and the opponent is modeled by two linear (probabilistic) functions fL(r)=a1r+b1 and fP(r)=a2r+b2, specifying the probability that a lob (and a pass, respectively) will win a point when the proportion of lobs played in the past trials is r. We measure the performance of a player in this model by his or her expected regret, namely how many fewer points the player expects to win as compared to the ideal player (one that knows the two probabilistic functions) as a function of t, the total number of trials, which is unknown to the player a priori. Assuming that the probabilistic functions satisfy the “matching shoulders condition,” i.e., fL(0)=fP(1), we obtain a variety of upper bounds for assumptions and restrictions of varying degrees, ranging from O(logt), O(t1/2), O(t3/5), O(t2/3) to O(t5/7) as well as a matching lower bound of order Ω(logt) for the first case. When the total number of trials t is given to the player in advance, the upper bounds can be improved significantly. An extended abstract describing part of this work has appeared in N. Abe and J. Takeuchi, 1993, in “Proceedings of the Sixth Annual ACM Workshop on Computational Learning Theory,” pp. 422–428.

Sampling Boolean Functions over Abelian Groups and Applications

We obtain efficient sampling methods for recovering or compressing functions over finite Abelian groups with few Fourier coefficients, i.e., functions that are (approximable by) linear combinations of few, possibly unknown Fourier basis functions or characters. Furthermore, our emphasis is on efficiently and deterministically finding small, uniform sample sets, which can be used for sampling all functions in natural approximation classes of Boolean functions. Due to this requirement, even the simplest versions of this problem (say, when the set of approximating characters is known) require somewhat different techniques from the character theory of finite Abelian groups that are commonly used in other discrete Fourier transform applications. We briefly discuss applications of our efficient, uniform sampling methods in computational learning theory, efficient generation of pseudorandom strings, and testing linearity; we also state highly related open problems that are not only applicable in these contexts, but are also of independent mathematical interest.

Function Learning from Interpolation

In this paper, we study a statistical property of classes of real-valued functions that we call approximation from interpolated examples. We derive a characterization of function classes that have this property, in terms of their ‘fat-shattering function’, a notion that has proved useful in computational learning theory. The property is central to a problem of learning real-valued functions from random examples in which we require satisfactory performance from every algorithm that returns a function which approximately interpolates the training examples.

Remarks on computational learning theory

Some remarks are given on the history, the main results and the future research directions of computational learning theory.

10.1162/153244303765208395

A fundamental open problem in computational learning theory is whether there is an attribute efficient learning algorithm for the concept class of decision lists (Rivest, 1987; Blum, 1996). We consider a weaker problem, where the concept class is restricted to decision lists with D alternations. For this class, we present a novel online algorithm that achieves a mistake bound of O(rDlog n), where r is the number of relevant variables, and n is the total number of variables. The algorithm can be viewed as a strict generalization of the famous Winnow algorithm by Littlestone (1988), and improves the O(r2Dlog n) mistake bound of Balanced Winnow. Our bound is stronger than a similar PAC-learning result of Dhagat and Hellerstein (1994). A combination of our algorithm with the algorithm suggested by Rivest (1987) might achieve even better bounds.

On-line Learning and the Metrical Task System Problem

The problem of combining expert advice, studied extensively in the Computational Learning Theory literature, and the Metrical Task System (MTS) problem, studied extensively in the area of On-line Algorithms, contain a number of interesting similarities. In this paper we explore the relationship between these problems and show how algorithms designed for each can be used to achieve good bounds and new approaches for solving the other. Specific contributions of this paper include: • An analysis of how two recent algorithms for the MTS problem can be applied to the problem of tracking the best expert in the “decision-theoretic” setting, providing good bounds and an approach of a much different flavor from the well-known multiplicative-update algorithms. • An analysis showing how the standard randomized Weighted Majority (or Hedge) algorithm can be used for the problem of “combining on-line algorithms on-line”, giving much stronger guarantees than the results of Azar, Y., Broder, A., & Manasse, M. (1993). Proc ACM-SIAM Symposium on Discrete Algorithms (pp. 432–440) when the algorithms being combined occupy a state space of bounded diameter. • A generalization of the above, showing how (a simplified version of) Herbster and Warmuth's weight-sharing algorithm can be applied to give a “finely competitive” bound for the uniform-space Metrical Task System problem. We also give a new, simpler algorithm for tracking experts, which unfortunately does not carry over to the MTS problem. Finally, we present an experimental comparison of how these algorithms perform on a process migration problem, a problem that combines aspects of both the experts-tracking and MTS formalisms.

Multiscale Annealing for Grouping and Unsupervised Texture Segmentation

We derive real-time global optimization methods for several clustering optimization problems commonly used in unsupervised texture segmentation. Speed is achieved by exploiting the image neighborhood relation of features to design a multiscale optimization technique, while accuracy and global optimization properties are gained using annealing techniques. Coarse grained cost functions are derived for central and histogram-based clustering as well as several sparse proximity-based clustering methods. For optimization deterministic annealing algorithms are applied. Annealing schedule, coarse-to-fine optimization and the estimated number of segments are tightly coupled by a statistical convergence criterion derived from computational learning theory. The notion of optimization scale parametrized by a computational temperature is thus unified with the scales defined by the image resolution and the model or segment complexity. The algorithms are benchmarked on Brodatz-like microtexture mixtures. Results are presented for an autonomous robotics application. Extensions are discussed in the context of prestructuring large image databases valuable for fast and reliable image retrieval.

The PAC-learnability of planning algorithms: Investigating simple planning domains

A framework was presented by Natarajan (B.K. Natarajan, On learning from exercises, in: Computational Learning Theory: Proceedings of the Second Annual Workshop, Morgan Kaufmann, Santa Cruz, CA, 1989, pp. 72–87) for the speedup learning of search algorithms. Speedup learning solves the problem of using teacher-provided examples to guide the improvement of algorithm efficiency. In this paper we extend their results by identifying two classes, the emonomial and emonotonic domains. We show that, in the Natarajan model, the speedup learning problem is solvable for the latter but not for the former. We outline further promising lines of investigation that may lead to a more comprehensive classification of the difficulty of speedup learning over a range of domains.

Synthesizers and Their Application to the Parallel Construction of Pseudo-Random Functions

A pseudo-random function is a fundamental cryptographic primitive that is essential for encryption, identification, and authentication. We present a new cryptographic primitive called pseudo-random synthesizer and show how to use it in order to get a parallel construction of a pseudo-random function. We show severalNC1implementations of synthesizers based on concrete intractability assumptions as factoring and the Diffie–Hellman assumption. This yields the first parallel pseudo-random functions (based on standard intractability assumptions) and the only alternative to the original construction of Goldreich, Goldwasser, and Micali. In addition, we show parallel constructions of synthesizers based on other primitives such as weak pseudo-random functions or trapdoor one-way permutations. The security of all our constructions is similar to the security of the underlying assumptions. The connection with problems in computational learning theory is discussed.

Introducing the Special Issue of Machine Learning Selected from PapersPresented at the 1997 Conference on Computational Learning Theory, COLT‘97

Introducing the Special Issue of Machine Learning Selected from PapersPresented at the 1997 Conference on Computational Learning Theory, COLT‘97

VC-Dimension Analysis of Object Recognition Tasks

We analyze the amount of data needed to carry out various model-based recognition tasks in the context of a probabilistic data collection model. We focus on objects that may be described as semi-algebraic subsets of a Euclidean space. This is a very rich class that includes polynomially described bodies, as well as polygonal objects, as special cases. The class of object transformations considered is wide, and includes perspective and affine transformations of 2D objects, and perspective projections of 3D objects. We derive upper bounds on the number of data features (associated with non-zero spatial error) which provably suffice for drawing reliable conclusions. Our bounds are based on a quantitative analysis of the complexity of the hypotheses class that one has to choose from. Our central tool is the VC-dimension, which is a well-studied parameter measuring the combinatorial complexity of families of sets. It turns out that these bounds grow linearly with the task complexity, measured via the VC-dimension of the class of objects one deals with. We show that this VC-dimension is at most logarithmic in the algebraic complexity of the objects and in the cardinality of the model library. Our approach borrows from computational learning theory. Both learning and recognition use evidence to infer hypotheses but as far as we know, their similarity was not exploited previously. We draw close relations between recognition tasks and a certain learnability framework and then apply basic techniques of learnability theory to derive our sample size upper bounds. We believe that other relations between learning procedures and visual tasks exist and hope that this work will trigger further fruitful study along these lines.

Vapnik-Chervonenkis dimension of recurrent neural networks

Most of the work on the Vapnik-Chervonenkis dimension of neural networks has been focused on feedforward networks. However, recurrent networks are also widely used in learning applications, in particular when time is a relevant parameter. This paper provides lower and upper bounds for the VC dimension of such networks. Several types of activation functions are discussed, including threshold, polynomial, piecewise-polynomial and sigmoidal functions. The bounds depend on two independent parameters: the number w of weights in the network, and the length k of the input sequence. In contrast, for feedforward networks, VC dimension bounds can be expressed as a function of w only. An important difference between recurrent and feedforward nets is that a fixed recurrent net can receive inputs of arbitrary length. Therefore we are particularly interested in the case k ⪢ w. Ignoring multiplicative constants, the main results say roughly the following: • • For architectures with activation (σ = any fixed nonlinear polynomial, the VC dimension is ≈ wk. • • For architectures with activation σ = any fixed piecewise polynomial, the VC dimension is between wk and w 2 k. • • For architectures with activation σ= H (threshold nets), the VC dimension is between w log( k w ) and min{ wk log wk, w 2 + w log wk}. • • For the standard sigmoid σ(x)= 1 (1 + e −x) , the VC dimension is between wk and w 4 k 2. An earlier version of this paper has appeared in Proc. 3rd European Workshop on Computational Learning Theory, Lecture Notes in Computer Science vol. 1208, Springer, Berlin, 1997, pp. 223–237.

A decision-theoretic extension of stochastic complexity and its applications to learning

Rissanen (1978) has introduced stochastic complexity to define the amount of information in a given data sequence relative to a given hypothesis class of probability densities, where the information is measured in terms of the logarithmic loss associated with universal data compression. This paper introduces the notion of extended stochastic complexity (ESC) and demonstrates its effectiveness in design and analysis of learning algorithms in on-line prediction and batch-learning scenarios. ESC can be thought of as an extension of Rissanen's stochastic complexity to the decision-theoretic setting where a general real-valued function is used as a hypothesis and a general loss function is used as a distortion measure. As an application of ESC to on-line prediction, this paper shows that a sequential realization of ESC produces an on-line prediction algorithm called Vovk's aggregating strategy, which can be thought of as an extension of the Bayes algorithm. We derive upper bounds on the cumulative loss for the aggregating strategy both of an expected form and a worst case form in the case where the hypothesis class is continuous. As an application of ESC to batch-learning, this paper shows that a batch-approximation of ESC induces a batch-learning algorithm called the minimum L-complexity algorithm (MLC), which is an extension of the minimum description length (MDL) principle. We derive upper bounds on the statistical risk for the MLC, which are the least to date. Through the ESC we give a unifying view of the most effective learning algorithms that have been explored in computational learning theory.

Support vector machines

My first exposure to Support Vector Machines came this spring when heard Sue Dumais present impressive results on text categorization using this analysis technique. This issue's collection of essays should help familiarize our readers with this interesting new racehorse in the Machine Learning stable. Bernhard Scholkopf, in an introductory overview, points out that a particular advantage of SVMs over other learning algorithms is that it can be analyzed theoretically using concepts from computational learning theory, and at the same time can achieve good performance when applied to real problems. Examples of these real-world applications are provided by Sue Dumais, who describes the aforementioned text-categorization problem, yielding the best results to date on the Reuters collection, and Edgar Osuna, who presents strong results on application to face detection. Our fourth author, John Platt, gives us a practical guide and a new technique for implementing the algorithm efficiently.

Specification and Simulation of Statistical Query Algorithms for Efficiency and Noise Tolerance

A recent innovation in computational learning theory is the statistical query (SQ) model. The advantage of specifying learning algorithms in this model is that SQ algorithms can be simulated in the probably approximately correct (PAC) model, both in the absenceandin the presence of noise. However, simulations of SQ algorithms in the PAC model have non-optimal time and sample complexities. In this paper, we introduce a new method for specifying statistical query algorithms based on a type ofrelative errorand provide simulations in the noise-free and noise-tolerant PAC models which yield more efficient algorithms. Requests for estimates of statistics in this new model take the following form: “Return an estimate of the statistic within a 1±μfactor, or return ⊥, promising that the statistic is less thanθ.” In addition to showing that this is a very natural language for specifying learning algorithms, we also show that this new specification is polynomially equivalent to standard SQ, and thus, known learnability and hardness results for statistical query learning are preserved. We then give highly efficient PAC simulations of relative error SQ algorithms. We show that the learning algorithms obtained by simulating efficient relative error SQ algorithms both in the absence of noise and in the presence of malicious noise have roughly optimal sample complexity. We also show that the simulation of efficient relative error SQ algorithms in the presence of classification noise yields learning algorithms at least as efficient as those obtained through standard methods, and in some cases improved, roughly optimal results are achieved. The sample complexities for all of these simulations are based on thedνmetric, which is a type of relative error metric useful for quantities which are small or even zero. We show that uniform convergence with respect to thedνmetric yields “uniform convergence” with respect to (μ, θ) accuracy. Finally, while we show that manyspecificlearning algorithms can be written as highly efficient relative error SQ algorithms, we also show, in fact, thatallSQ algorithms can be written efficiently by proving general upper bounds on the complexity of (μ, θ) queries as a function of the accuracy parameterε. As a consequence of this result, we give general upper bounds on the complexity of learning algorithms achieved through the use of relative error SQ algorithms and the simulations described above.

The sample complexity of pattern classification with neural networks: the size of the weights is more important than the size of the network

Sample complexity results from computational learning theory, when applied to neural network learning for pattern classification problems, suggest that for good generalization performance the number of training examples should grow at least linearly with the number of adjustable parameters in the network. Results in this paper show that if a large neural network is used for a pattern classification problem and the learning algorithm finds a network with small weights that has small squared error on the training patterns, then the generalization performance depends on the size of the weights rather than the number of weights. For example, consider a two-layer feedforward network of sigmoid units, in which the sum of the magnitudes of the weights associated with each unit is bounded by A and the input dimension is n. We show that the misclassification probability is no more than a certain error estimate (that is related to squared error on the training set) plus A/sup 3/ /spl radic/((log n)/m) (ignoring log A and log m factors), where m is the number of training patterns. This may explain the generalization performance of neural networks, particularly when the number of training examples is considerably smaller than the number of weights. It also supports heuristics (such as weight decay and early stopping) that attempt to keep the weights small during training. The proof techniques appear to be useful for the analysis of other pattern classifiers: when the input domain is a totally bounded metric space, we use the same approach to give upper bounds on misclassification probability for classifiers with decision boundaries that are far from the training examples.

Flexible features, connectionism, and computational learning theory

This commentary is an elaboration on Schyns, Goldstone & Thibaut's proposal for flexible features in categorization in the light of three areas not explicitly discussed by the authors: connectionist models of categorization, computational learning theory, and constructivist theories of the mind. In general, the authors' proposal is strongly supported, paving the way for model extensions and for interesting novel cognitive research. Nor is the authors' proposal incompatible with theories positing some fixed set of features.