Probably Approximately Correct Research Articles

Circuit lower bounds from learning-theoretic approaches

An important open problem in computational complexity theory is to prove the size of circuits, namely, Boolean circuit lower bounds, necessary to solve explicit problems. We survey learning-theoretic approaches to proofs of Boolean circuit lower bounds in this paper. In particular, we discuss how to prove circuit lower bounds in uniform classes by assuming (or constructing) circuit-learning algorithms in several settings, such as the exact, probably approximately correct (PAC), and statistical query learning models.

Data science as a language: challenges for computer science\u2014a position paper

In this paper, I posit that from a research point of view, Data Science is a language. More precisely Data Science is doing Science using computer science as a language for datafied sciences; much as mathematics is the language of, e.g., physics. From this viewpoint, three (classes) of challenges for computer science are identified; complementing the challenges the closely related Big Data problem already poses to computer science. I discuss the challenges with references to, in my opinion, related, interesting directions in computer science research; note, I claim neither that these directions are the most appropriate to solve the challenges nor that the cited references represent the best work in their field, they are inspirational to me. So, what are these challenges? Firstly, if computer science is to be a language, what should that language look like? While our traditional specifications such as pseudocode are an excellent way to convey what has been done, they fail for more mathematics like reasoning about computations. Secondly, if computer science is to function as a foundation of other, datafied, sciences, its own foundations should be in order. While we have excellent foundations for supervised learning—e.g., by having loss functions to optimize and, more general, by PAC learning (Valiant in Commun ACM 27(11):1134–1142, 1984)—this is far less true for unsupervised learning. Kolmogorov complexity—or, more general, Algorithmic Information Theory—provides a solid base (Li and Vitányi in An introduction to Kolmogorov complexity and its applications, Springer, Berlin, 1993). It provides an objective criterion to choose between competing hypotheses, but it lacks, e.g., an objective measure of the uncertainty of a discovery that datafied sciences need. Thirdly, datafied sciences come with new conceptual challenges. Data-driven scientists come up with data analysis questions that sometimes do and sometimes don’t, fit our conceptual toolkit. Clearly, computer science does not suffer from a lack of interesting, deep, research problems. However, the challenges posed by data science point to a large reservoir of untapped problems. Interesting, stimulating problems, not in the least because they are posed by our colleagues in datafied sciences. It is an exciting time to be a computer scientist.

Guest Column

This paper surveys quantum learning theory: the theoretical aspects of machine learning using quantum computers. We describe the main results known for three models of learning: exact learning from membership queries, and Probably Approximately Correct (PAC) and agnostic learning from classical or quantum examples.

Ensemble method to joint inference for knowledge extraction

Traditionally, the probably approximately correct (PAC) learning refers the single concept class. We discuss the PAC framework of the multiple tasks in the joint inference model. And we extend PAC learning to multi-concept classes.We present an ensemble learning approach to joint inference on the three NLP sub-tasks. We explain how to combine those weak learners to a strong ability and present the dynamic weighted combination method in the ensemble joint inference model.Our Ensemble Markov Logic Networks (EMLNs) address the problem of the Markov Logic Networks intractable dealing with the large scale data. Experiments show that this approach leads to a higher precision and recall than that of pipeline approaches. Joint inference is a fundamental issue in the field of artificial intelligence. The greatest advantage of the joint inference is demonstrated by its capability of avoiding errors from cascading and accumulating on a pipeline of multiple chained sub-tasks. Markov Logic Network(MLN) is the most common joint inference model that provides a flexible representation and handles uncertainty. It has been applied successfully to joint inference on many natural language processing tasks to avoid error propagation. However, due to the great expressiveness of first-order logic, the representation for it in MLN generates rather complicated graph structures, which makes the learning and inference on large scale data intractably. In this paper, we present an ensemble learning approach to deal with the challenges in MLNs. Firstly, we give a proof within the probably approximately correct (PAC) framework. The proof points out what conditions are necessary for successful applying the ensemble learning approach to MLN. Secondly, the paper explains how to combine the learners. Finally, in order to illustrate the working mechanism of the ensemble joint inference model, we present an Ensemble Markov Logic Networks (EMLNs) method and use it to extract knowledge from a large scale corpus published by Google.11code.google.com/p/relation-extraction-corpus/. Experiments suggest that significant speedup can be gained by the EMLNs. Meanwhile, it show that this approach leads to a higher precision and recall than that of those pipeline approaches.

A Lockdown Technique to Prevent Machine Learning on PUFs for Lightweight Authentication

We present a lightweight PUF-based authentication approach that is practical in settings where a server authenticates a device, and for use cases where the number of authentications is limited over a device's lifetime. Our scheme uses a server-managed challenge/response pair (CRP) lockdown protocol: unlike prior approaches, an adaptive chosen-challenge adversary with machine learning capabilities cannot obtain new CRPs without the server's implicit permission. The adversary is faced with the problem of deriving a PUF model with a limited amount of machine learning training data. Our system-level approach allows a so-called strong PUF to be used for lightweight authentication in a manner that is heuristically secure against today's best machine learning methods through a worst-case CRP exposure algorithmic validation. We also present a degenerate instantiation using a weak PUF that is secure against computationally unrestricted adversaries, which includes any learning adversary, for practical device lifetimes and read-out rates. We validate our approach using silicon PUF data, and demonstrate the feasibility of supporting 10, 1,000, and 1M authentications, including practical configurations that are not learnable with polynomial resources, e.g., the number of CRPs and the attack runtime, using recent results based on the probably-approximately-correct (PAC) complexity-theoretic framework.

A theoretical framework for deep transfer learning

We generalize the notion of PAC learning to include transfer learning. In our framework, the linkage between the source and the target tasks is a result of having the sample distribution of all classes drawn from the same distribution of distributions, and by restricting all source and a target concepts to belong to the same hypothesis subclass. We have two models: an adversary model and a randomized model. In the adversary model, we show that for binary classification, conventional PAC-learning is equivalent to the new notion of PAC-transfer and to transfer generalization of the VC-dimension. For regression, we show that PAC-transferability may exist even in the absence of PAC-learning. In both adversary and randomized models, we provide PAC-Bayesian and VC-style generalization bounds to transfer learning. In the randomized model, we provide bounds specifically derived for Deep Learning. A wide discussion on the tradeoffs between the different involved parameters in the bounds is provided. We demonstrate both cases in which transfer does not reduce the sample size (‘trivial transfer’) and cases in which the sample size is reduced (‘non-trivial transfer’).

PAC learning of arbiter PUFs

The general concept of physically unclonable functions (PUFs) has been nowadays widely accepted and adopted to meet the requirements of secure identification and key generation/storage for cryptographic ciphers. However, shattered by different attacks, e.g., modeling attacks, it has been proved that the promised security features of arbiter PUFs, including unclonability and unpredictability, are not supported unconditionally. However, so far the success of existing modeling attacks relies on pure trial and error estimates. This means that neither the probability of obtaining a useful model (confidence), nor the sufficient number of CRPs, nor the probability of correct prediction (accuracy) is guaranteed. To address these issues, this work presents a probably approximately correct (PAC) learning algorithm. Based on a crucial discretization process, we are able to define a Deterministic finite automaton (of polynomial size), which exactly accepts the regular language corresponding to the challenges mapped by the given PUF to one responses.

The optimal sample complexity OF PAC learning

This work establishes a new upper bound on the number of samples sufficient for PAC learning in the realizable case. The bound matches known lower bounds up to numerical constant factors. This solv...

On the Sample Complexity of Cancer Pathways Identification.

Advances in DNA sequencing technologies have enabled large cancer sequencing studies, collecting somatic mutation data from a large number of cancer patients. One of the main goals of these studies is the identification of all cancer genes--genes associated with cancer. Its achievement is complicated by the extensive mutational heterogeneity of cancer, due to the fact that important mutations in cancer target combinations of genes (i.e., pathways). Recently, the pattern of mutual exclusivity among mutations in a cancer pathway has been observed, and methods that find significant combinations of cancer genes by detecting mutual exclusivity have been proposed. A key question in the analysis of mutual exclusivity is the computation of the minimum number of samples required to reliably find a meaningful set of mutually exclusive mutations in the data, or conclude that there is no such set. In general, the problem of determining the sample complexity, or the number of samples required to identify significant combinations of features, of genomic problems is largely unexplored. In this work we propose a framework to analyze the sample complexity of problems that arise in the study of genomic datasets. Our framework is based on tools from combinatorial analysis and statistical learning theory that have been used for the analysis of machine learning and probably approximately correct (PAC) learning. We use our framework to analyze the problem of the identification of cancer pathways through mutual exclusivity analysis. We analytically derive matching upper and lower bounds on the sample complexity of the problem, showing that sample sizes much larger than currently available may be required to identify all the cancer genes in a pathway. We also provide two algorithms to find a cancer pathway from a large genomic dataset. On simulated and cancer data, we show that our algorithms can be used to identify cancer pathways from large genomic datasets.

Probabilistic learnability of context-free grammars with basic distributional properties from positive examples

In recent years different interesting subclasses of cfls have been found to be learnable by techniques generically called distributional learning. The theoretical study on the exact learning of cfls by those techniques under different learning schemes is now quite mature. On the other hand, positive results on the pac learnability of cfls are rather limited and quite weak. This paper shows that several subclasses of context-free languages that are known to be exactly learnable with positive data and membership queries by distributional learning techniques are pac learnable from positive data under some assumptions on the string distribution.

PAC learning, VC dimension, and the arithmetic hierarchy

We compute that the index set of PAC-learnable concept classes is m-complete $${\Sigma^{0}_{3}}$$Σ30 within the set of indices for all concept classes of a reasonable form. All concept classes considered are computable enumerations of computable $${\Pi^{0}_{1}}$$?10 classes, in a sense made precise here. This family of concept classes is sufficient to cover all standard examples, and also has the property that PAC learnability is equivalent to finite VC dimension.

Citation data as a proxy for quality or scientific influence are at best PAC (probably approximately correct)

In this communication I give a brief introduction to Valiant's probably approximately correct (PAC) theory, provide an extension that goes beyond Valiant's ideas (and beyond the domain for which this theory was meant), and come to an interpretation in terms of research evaluation. As such, PAC provides a framework for a theory of research evaluation.

A probabilistic approach to case-based inference

The central problem in case based reasoning (CBR) is to infer a solution for a new problem-instance by using a collection of existing problem–solution cases. The basic heuristic guiding CBR is the hypothesis that similar problems have similar solutions. Recently, some attempts at formalizing CBR in a theoretical framework have been made, including work by Hüllermeier who established a link between CBR and the probably approximately correct (PAC) theoretical model of learning in his ‘case-based inference’ (CBI) formulation. In this paper we develop further such probabilistic modelling, framing CBI it as a multi-category classification problem. We use a recently-developed notion of geometric margin of classification to obtain generalization error bounds.

MEC--a near-optimal online reinforcement learning algorithm for continuous deterministic systems.

In this paper, the first probably approximately correct (PAC) algorithm for continuous deterministic systems without relying on any system dynamics is proposed. It combines the state aggregation technique and the efficient exploration principle, and makes high utilization of online observed samples. We use a grid to partition the continuous state space into different cells to save samples. A near-upper Q operator is defined to produce a near-upper Q function using samples in each cell. The corresponding greedy policy effectively balances between exploration and exploitation. With the rigorous analysis, we prove that there is a polynomial time bound of executing nonoptimal actions in our algorithm. After finite steps, the final policy reaches near optimal in the framework of PAC. The implementation requires no knowledge of systems and has less computation complexity. Simulation studies confirm that it is a better performance than other similar PAC algorithms.

Probably Approximately Correct, A Book Review

Probably Approximately Correct, A Book Review

A data-based online reinforcement learning algorithm satisfying probably approximately correct principle

This paper proposes a probably approximately correct (PAC) algorithm that directly utilizes online data efficiently to solve the optimal control problem of continuous deterministic systems without system parameters for the first time. The dependence on some specific approximation structures is crucial to limit the wide application of online reinforcement learning (RL) algorithms. We utilize the online data directly with the kd-tree technique to remove this limitation. Moreover, we design the algorithm in the PAC principle. Complete theoretical proofs are presented, and three examples are simulated to verify its good performance. It draws the conclusion that the proposed RL algorithm specifies the maximum running time to reach a near-optimal control policy with only online data.

A machine learning approach to create blocking criteria for record linkage.

Record linkage, a part of data cleaning, is recognized as one of most expensive steps in data warehousing. Most record linkage (RL) systems employ a strategy of using blocking filters to reduce the number of pairs to be matched. A blocking filter consists of a number of blocking criteria. Until recently, blocking criteria are selected manually by domain experts. This paper proposes a new method to automatically learn efficient blocking criteria for record linkage. Our method addresses the lack of sufficient labeled data for training. Unlike previous works, we do not consider a blocking filter in isolation but in the context of an accompanying matcher which is employed after the blocking filter. We show that given such a matcher, the labels (assigned to record pairs) that are relevant for learning are the labels assigned by the matcher (link/nonlink), not the labels assigned objectively (match/unmatch). This conclusion allows us to generate an unlimited amount of labeled data for training. We formulate the problem of learning a blocking filter as a Disjunctive Normal Form (DNF) learning problem and use the Probably Approximately Correct (PAC) learning theory to guide the development of algorithm to search for blocking filters. We test the algorithm on a real patient master file of 2.18 million records. The experimental results show that compared with filters obtained by educated guess, the optimal learned filters have comparable recall but reduce throughput (runtime) by an order-of-magnitude factor.

A Generalized Model of PAC Learning and its Applicability

We combine a new data model, where the random classification is subjected to rather weak restrictions which in turn are based on the Mammen−Tsybakov [E. Mammen and A.B. Tsybakov, Ann. Statis. 27 (1999) 1808–1829; A.B. Tsybakov, Ann. Statis. 32 (2004) 135–166.] small margin conditions, and the statistical query (SQ) model due to Kearns [M.J. Kearns, J. ACM 45 (1998) 983–1006] to what we refer to as PAC + SQ model. We generalize the class conditional constant noise (CCCN) model introduced by Decatur [S.E. Decatur, in ICML ’97: Proc. of the Fourteenth Int. Conf. on Machine Learn. Morgan Kaufmann Publishers Inc. San Francisco, CA, USA (1997) 83–91] to the noise model orthogonal to a set of query functions. We show that every polynomial time PAC + SQ learning algorithm can be efficiently simulated provided that the random noise rate is orthogonal to the query functions used by the algorithm given the target concept. Furthermore, we extend the constant-partition classification noise (CPCN) model due to Decatur [S.E. Decatur, in ICML ’97: Proc. of the Fourteenth Int. Conf. on Machine Learn. Morgan Kaufmann Publishers Inc. San Francisco, CA, USA (1997) 83–91] to what we call the constant-partition piecewise orthogonal (CPPO) noise model. We show how statistical queries can be simulated in the CPPO scenario, given the partition is known to the learner. We show how to practically use PAC + SQ simulators in the noise model orthogonal to the query space by presenting two examples from bioinformatics and software engineering. This way, we demonstrate that our new noise model is realistic.

Learning schema mappings

A schema mapping is a high-level specification of the relationship between a source schema and a target schema. Recently, a line of research has emerged that aims at deriving schema mappings automatically or semi-automatically with the help of data examples, that is, pairs consisting of a source instance and a target instance that depict, in some precise sense, the intended behavior of the schema mapping. Several different uses of data examples for deriving, refining, or illustrating a schema mapping have already been proposed and studied. In this article, we use the lens of computational learning theory to systematically investigate the problem of obtaining algorithmically a schema mapping from data examples. Our aim is to leverage the rich body of work on learning theory in order to develop a framework for exploring the power and the limitations of the various algorithmic methods for obtaining schema mappings from data examples. We focus on GAV schema mappings, that is, schema mappings specified by GAV (Global-As-View) constraints. GAV constraints are the most basic and the most widely supported language for specifying schema mappings. We present an efficient algorithm for learning GAV schema mappings using Angluin's model of exact learning with membership and equivalence queries. This is optimal, since we show that neither membership queries nor equivalence queries suffice, unless the source schema consists of unary relations only. We also obtain results concerning the learnability of schema mappings in the context of Valiant's well-known PAC (Probably-Approximately-Correct) learning model, and concerning the learnability of restricted classes of GAV schema mappings. Finally, as a byproduct of our work, we show that there is no efficient algorithm for approximating the shortest GAV schema mapping fitting a given set of examples, unless the source schema consists of unary relations only.

Bounds on the sample complexity for private learning and private data release

Learning is a task that generalizes many of the analyses that are applied to collections of data, in particular, to collections of sensitive individual information. Hence, it is natural to ask what can be learned while preserving individual privacy. Kasiviswanathan et al. (in SIAM J. Comput., 40(3):793---826, 2011) initiated such a discussion. They formalized the notion of private learning, as a combination of PAC learning and differential privacy, and investigated what concept classes can be learned privately. Somewhat surprisingly, they showed that for finite, discrete domains (ignoring time complexity), every PAC learning task could be performed privately with polynomially many labeled examples; in many natural cases this could even be done in polynomial time. While these results seem to equate non-private and private learning, there is still a significant gap: the sample complexity of (non-private) PAC learning is crisply characterized in terms of the VC-dimension of the concept class, whereas this relationship is lost in the constructions of private learners, which exhibit, generally, a higher sample complexity. Looking into this gap, we examine several private learning tasks and give tight bounds on their sample complexity. In particular, we show strong separations between sample complexities of proper and improper private learners (such separation does not exist for non-private learners), and between sample complexities of efficient and inefficient proper private learners. Our results show that VC-dimension is not the right measure for characterizing the sample complexity of proper private learning. We also examine the task of private data release (as initiated by Blum et al. in STOC, pp. 609---618, 2008), and give new lower bounds on the sample complexity. Our results show that the logarithmic dependence on size of the instance space is essential for private data release.