Polynomial Time Learning Algorithm Research Articles

On Cost-Efficient Learning of Data Dependency

In this paper, we consider the problem of learning a tree graph structure that represents the statistical data dependency among nodes for a set of data samples generated by nodes, which provides the basic structure to perform a probabilistic inference task. Inference in the data graph includes marginal inference and maximum a posteriori (MAP) estimation, and belief propagation (BP) is a commonly used algorithm to compute the marginal distribution of nodes via message-passing, incurring non-negligible amount of communication cost. We inevitably have the trade-off between the inference accuracy and the message-passing cost because the learned structure of data dependency and physical connectivity graph are often highly different. In this paper, we formalize this trade-off in an optimization problem which outputs the data dependency graph that jointly considers learning accuracy and message-passing costs. We focus on two popular implementations of BP, <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ASYNC-BP</monospace> and <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">SYNC-BP</monospace> , which have different message-passing mechanisms and cost structures. In <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ASYNC-BP</monospace> , we propose a polynomial-time learning algorithm that is optimal, motivated by finding a maximum weight spanning tree of a complete graph. In <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">SYNC-BP</monospace> , we prove the NP-hardness of the problem and propose a greedy heuristic. For both BP implementations, we quantify how the error probability that the learned cost-efficient data graph differs from the ideal one decays as the number of data samples grows, using the large deviation principle, which provides a guideline on how many samples are necessary to obtain a certain trade-off. We validate our theoretical findings through extensive simulations, which confirms that it has a good match.

Hierarchical semi-Markov conditional random fields for deep recursive sequential data

We present the hierarchical semi-Markov conditional random field (HSCRF), a generalisation of linear-chain conditional random fields to model deep nested Markov processes. It is parameterised as a conditional log-linear model and has polynomial time algorithms for learning and inference. We derive algorithms for partially-supervised learning and constrained inference. We develop numerical scaling procedures that handle the overflow problem. We show that when depth is two, the HSCRF can be reduced to the semi-Markov conditional random fields. Finally, we demonstrate the HSCRF on two applications: (i) recognising human activities of daily living (ADLs) from indoor surveillance cameras, and (ii) noun-phrase chunking. The HSCRF is capable of learning rich hierarchical models with reasonable accuracy in both fully and partially observed data cases.

Efficient learning of multiple context-free languages with multidimensional substitutability from positive data

Recently Clark and Eyraud (2007) [10] have shown that substitutable context-free languages, which capture an aspect of natural language phenomena, are efficiently identifiable in the limit from positive data. Generalizing their work, this paper presents a polynomial-time learning algorithm for new subclasses of multiple context-free languages with variants of substitutability.

Learning of Elementary Formal Systems with Two Clauses Using Queries

An elementary formal system, EFS for short, is a kind of logic program over strings, and regarded as a set of rules to generate a language. For an EFS Γ, the language L(Γ) denotes the set of all strings generated by Γ. We consider a new form of EFS, called a restricted two-clause EFS, and denote by $r{\\cal EF\\!S}$ the set of all restricted two-clause EFSs. Then we study the learnability of $r{\\cal EF\\!S}$ in the exact learning model. The class $r{\\cal EF\\!S}$ contains the class of regular patterns, which is extensively studied in Learning Theory. Let Γ* be a target EFS in $r{\\cal EF\\!S}$ of learning. In the exact learning model, an oracle for superset queries answers “yes” for an input EFS Γ in $r{\\cal EF\\!S}$ if L(Γ) is a superset of L(Γ*), and outputs a string in L(Γ*) - L(Γ), otherwise. An oracle for membership queries answers “yes” for an input string w if w is included in L(Γ*), and answers “no”, otherwise. We show that any EFS in $r{\\cal EF\\!S}$ is exactly identifiable in polynomial time using membership and superset queries. Moreover, for other types of queries, we show that there exists no polynomial time learning algorithm for $r{\\cal EF\\!S}$ by using the queries. This result indicates the hardness of learning the class $r{\\cal EF\\!S}$ in the exact learning model, in general.

Learning of Finite Unions of Tree Patterns with Repeated Internal Structured Variables from Queries

The exact learning model by Angluin (1988) is a mathematical model of learning via queries in computational learning theory. A term tree is a tree pattern consisting of ordered tree structures and repeated structured variables, which occur more than once. Thus, a term tree is suited for representing common tree structures based on tree-structured data, such as HTML and XML files on the Web. In this paper, we consider the learnability of finite unions of term trees with repeated variables in the exact learning model. We present polynomial time learning algorithms for finite unions of term trees with repeated variables by using superset and restricted equivalence queries. Moreover, we show that there exists no polynomial time learning algorithm for finite unions of term trees by using restricted equivalence, membership, and subset queries. This result indicates the hardness of learning finite unions of term trees in the exact learning model.

On learning embedded midbit functions

A midbit function on ℓ binary inputs x 1 , … , x ℓ outputs the middle bit in the binary representation of x 1 + ⋯ + x ℓ . We consider the problem of Probably Approximately Correct (PAC) learning embedded midbit functions, where the set S ⊂ { x 1 , … , x n } of relevant variables on which the midbit depends is unknown to the learner. To motivate this problem, we first point out that a result of Green et al. implies that a polynomial time learning algorithm for the class of embedded midbit functions would immediately yield a fairly efficient (quasipolynomial time) (PAC) learning algorithm for the entire complexity class ACC. We then give two different subexponential learning algorithms, each of which learns embedded midbit functions under any probability distribution in 2 n log n time. Finally, we give a polynomial time algorithm for learning embedded midbit functions under the uniform distribution.

Polynomial Time Learnability of a Sub-class of Linear Languages

We propose some PAC like settings for a learning problem of a sub-class of linear languages, and show its polynomial time learnability in each of our settings. Here, the sub-class of linear languages is newly defined, and it includes the class of regular languages and the class of even linear languages. We show a polynomial time learning algorithm in either of the following settings with a fixed but unknown probability distribution for examples.(1) The first case is when the learner can use randomly drawn examples, membership queries, and a set of representative samples.(2) The second case is when the learner can use randomly drawn examples, membership queries, and both of the size of a grammar which can generate the target language and d. Where d is the probability such that the rarest rule in the target grammar occurs in the derivation of a randomly drawn example. In each case, for the target language Lt, the hypothesis Lhsatisfies thatPr[P(Lh Δ Lt) ≤ e] ≥ 1 - δ for the error parameter 0 < e ≤ 1 and the confidential parameter 0 < δ ≤ 1.

A near-optimal polynomial time algorithm for learning in certain classes of stochastic games

We present a new algorithm for polynomial time learning of optimal behavior in single-controller stochastic games. This algorithm incorporates and integrates important recent results of Kearns and Singh (Proc. ICML-98, 1998) in reinforcement learning and of Monderer and Tennenholtz (J. Artif. Intell. Res. 7, 1997, p. 231) in repeated games. In stochastic games, the agent must cope with the existence of an adversary whose actions can be arbitrary. In particular, this adversary can withhold information about the game matrix by refraining from (or rarely) performing certain actions. This forces upon us an exploration versus exploitation dilemma more complex than in Markov decision processes in which, given information about particular parts of a game matrix, the agent must decide how much effort to invest in learning the unknown parts of the matrix. We present a polynomial time algorithm that addresses these issues in the context of the class of single controller stochastic games, providing the agent with near-optimal return.

Malicious Omissions and Errors in Answers to Membership Queries

We consider two issues in polynomial-time exact learning of concepts using membership and equivalence queries: (1) errors or omissions in answers to membership queries, and (2) learning finite variants of concepts drawn from a learnable class.To study (1), we introduce two new kinds of membership queries: limited membership queries and malicious membership queries. Each is allowed to give incorrect responses on a maliciously chosen set of strings in the domain. Instead of answering correctly about a string, a limited membership query may give a special “I don't know” answer, while a malicious membership query may give the wrong answer. A new parameter Lis used to bound the length of an encoding of the set of strings that receive such incorrect answers. Equivalence queries are answered correctly, and learning algorithms are allowed time polynomial in the usual parameters and L. Any class of concepts learnable in polynomial time using equivalence and malicious membership queries is learnable in polynomial time using equivalence and limited membership queries; the converse is an open problem. For the classes of monotone monomials and monotone k-term DNF formulas, we present polynomial-time learning algorithms using limited membership queries alone. We present polynomial-time learning algorithms for the class of monotone DNF formulas using equivalence and limited membership queries, and using equivalence and malicious membership queries.To study (2), we consider classes of concepts that are polynomially closed under finite exceptions and a natural operation to add exception tables to a class of concepts. Applying this operation, we obtain the class of monotone DNF formulas with finite exceptions. We give a polynomial-time algorithm to learn the class of monotone DNF formulas with finite exceptions using equivalence and membership queries. We also give a general transformation showing that any class of concepts that is polynomially closed under finite exceptions and is learnable in polynomial time using standard membership and equivalence queries is also polynomial-time learnable using malicious membership and equivalence queries. Corollaries include the polynomial-time learnability of the following classes using malicious membership and equivalence queries: deterministic finite acceptors, boolean decision trees, and monotone DNF formulas with finite exceptions.

On the Fourier spectrum of monotone functions

In this paper, monotone Boolean functions are studied using harmonic analysis on the cube. The main result is that any monotone Boolean function has most of its power spectrum on its Fourier coefficients of degree at most O(√n) under any product distribution. This is similar to a result of Linial et al. [1993], which showed that AC 0 functions have almost all of their power spectrum on the coefficients of degree, at most (log n) O(1) , under the uniform distribution. As a consequence of the main result, the following two corollaries are obtained : -For any e > 0, monotone Boolean functions are PAC learnable with error e under product distributions in time 2 O((1/e) √ n) . -Any monotone Boolean function can be approximated within error e under product distributions by a non-monotone Boolean circuit of size 2 O(1/e √ n) and depth O(1/e √n). The learning algorithm runs in time subexponential as long as the required error is Ω(1/(√n log n)). It is shown that this is tight in the sense that for any subexponential time algorithm there is a monotone Boolean function for which this algorithm cannot approximate with error better than O(1/√n). The main result is also applied to other problems in learning and complexity theory. In learning theory, several polynomial-time algorithms for learning some classes of monotone Boolean functions, such as Boolean functions with O(log 2 n/log log n) relevant variables, are presented. In complexity theory, some questions regarding monotone NP-complete problems are addressed.

Efficient learning with equivalence queries of conjunctions of modulo functions

A modulo m function takes value 0 when the weighted sum of the input variables is congruent to 0 mod m and takes value 1 otherwise. In this work we show a polynomial time algorithm for learning with equivalence queries the class of conjunctions of modulo p functions on n variables, for any fixed prime p. The result is also extended to learning with bounded adversarial noise.

On the Learnability of Disjunctive Normal Form Formulas

We present two related results about the learnability of disjunctive normal form (DNF) formulas. First we show that a common approach for learning arbitrary DNF formulas requires exponential time. We then contrast this with a polynomial time algorithm for learning “most” (rather than all) DNF formulas. A natural approach for learning boolean functions involves greedily collecting the prime implicants of the hidden function. In a seminal paper of learning theory, Valiant demonstrated the efficacy of this approach for learning monotone DNF, and suggested this approach for learning DNF. Here we show that no algorithm using such an approach can learn DNF in polynomial time. We show this by constructing a counterexample DNF formula which would force such an algorithm to take exponential time. This counterexample seems to capture much of what makes DNF hard to learn, and thus is useful to consider when evaluating the run-time of a proposed DNF learning algorithm. This hardness result, as well as other hardness results for learning DNF, relies on the construction of particular hard-to-learn formulas, formulas that appear to be relatively rare. This raises the question of whether most DNF formulas are learnable. For certain natural definitions of “most DNF formulas,” we answer this question affirmatively.

Separating Distribution-Free and Mistake-Bound Learning Models over the Boolean Domain

Two of the most commonly used models in computational learning theory are the distribution-free model in which examples are chosen from a fixed but arbitrary distribution, and the absolute mistake-bound model in which examples are presented in an arbitrary order. Over the Boolean domain $\{ 0,1\} ^n $, it is known that if the learner is allowed unlimited computational resources then any concept class learnable in one model is also learnable in the other. In addition, any polynomial-time learning algorithm for a concept class in the mistake-bound model can be transformed into one that learns the class in the distribution-free model. This paper shows that if one-way functions exist, then the mistake-bound model is strictly harder than the distribution-free model for polynomial-time learning. Specifically, given a one-way function, it is shown how to create a concept class over $\{ 0,1\} ^n $ that is learnable in polynomial time in the distribution-free model, but not in the absolute mistake-bound model. In addition, the concept class remains hard to learn in the mistake-bound model even if the learner is allowed a polynomial number of membership queries. The concepts considered are based upon the Goldreich, Goldwasser, and Micali random function construction [Goldreich, Goldwasser, and Micali, Journal ACM, 33 (1986), pp. 792–807] and involve creating the following new cryptographic object: an exponentially long sequence of strings $\sigma _1 ,\sigma _2 , \ldots ,\sigma _r $ over $\{ 0,1\} ^n $ that is hard to compute in one direction (given $\sigma _i $ one cannot compute $\sigma _j $ for $j < i$) but is easy to compute and even make random-access jumps in the other direction (given $\sigma _i $ and $j > i$ one can compute $\sigma _j $, even if j is exponentially larger than i ). Similar sequences considered previously [Blum, Blum, and Shub, SIAM J. Comput., 15 (1986), pp. 364–383], [Blum and Micali, SIAM J. Comput., 13 (1984), pp. 850–863] did not allow random-access jumps forward without knowledge of a seed allowing one to compute backwards as well.

Cryptographic limitations on learning Boolean formulae and finite automata

In this paper, we prove the intractability of learning several classes of Boolean functions in the distribution-free model (also called the Probably Approximately Correct or PAC model) of learning from examples. These results are representation independent , in that they hold regardless of the syntactic form in which the learner chooses to represent its hypotheses. Our methods reduce the problems of cracking a number of well-known public-key cryptosystems to the learning problems. We prove that a polynomial-time learning algorithm for Boolean formulae, deterministic finite automata or constant-depth threshold circuits would have dramatic consequences for cryptography and number theory. In particular, such an algorithm could be used to break the RSA cryptosystem, factor Blum integers (composite numbers equivalent to 3 modulo 4), and detect quadratic residues. The results hold even if the learning algorithm is only required to obtain a slight advantage in prediction over random guessing. The techniques used demonstrate an interesting duality between learning and cryptography. We also apply our results to obtain strong intractability results for approximating a generalization of graph coloring.

On learning from queries and counterexamples in the presence of noise

Recently Angluin and Laird have introduced the classification noise process in the Valiant learnability model and proposed an interesting problem to explore the effect of noise in a situation that calls for queries as well as random sampling. In this paper, we present a general method to modify a polynomial-time learning algorithm from a sampling oracle and membership queries to compensate for random errors in the sampling and query responses.

Learning regular sets from queries and counterexamples

The problem of identifying an unknown regular set from examples of its members and nonmembers is addressed. It is assumed that the regular set is presented by a minimally adequate Teacher , which can answer membership queries about the set and can also test a conjecture and indicate whether it is equal to the unknown set and provide a counterexample if not. (A counterexample is a string in the symmetric difference of the correct set and the conjectured set.) A learning algorithm L ∗ is described that correctly learns any regular set from any minimally adequate Teacher in time polynomial in the number of states of the minimum dfa for the set and the maximum length of any counterexample provided by the Teacher. It is shown that in a stochastic setting the ability of the Teacher to test conjectures may be replaced by a random sampling oracle, EX ( ). A polynomial-time learning algorithm is shown for a particular problem of context-free language identification.