VC-dimension Research Articles

The complexity of exact learning of acyclic conditional preference networks from swap examples

Learning of user preferences, as represented by, for example, Conditional Preference Networks (CP-nets), has become a core issue in AI research. Recent studies investigate learning of CP-nets from randomly chosen examples or from membership and equivalence queries. To assess the optimality of learning algorithms as well as to better understand the combinatorial structure of classes of CP-nets, it is helpful to calculate certain learning-theoretic information complexity parameters. This article focuses on the frequently studied case of exact learning from so-called swap examples, which express preferences among objects that differ in only one attribute. It presents bounds on or exact values of some well-studied information complexity parameters, namely the VC dimension, the teaching dimension, and the recursive teaching dimension, for classes of acyclic CP-nets. We further provide algorithms that exactly learn tree-structured and general acyclic CP-nets from membership queries. Using our results on complexity parameters, we prove that our algorithms, as well as another query learning algorithm for acyclic CP-nets presented in the literature, are near-optimal.

Exact lower bounds for the agnostic probably-approximately-correct (PAC) machine learning model

We provide an exact nonasymptotic lower bound on the minimax expected excess risk (EER) in the agnostic probably-approximately-correct (PAC) machine learning classification model and identify minimax learning algorithms as certain maximally symmetric and minimally randomized “voting” procedures. Based on this result, an exact asymptotic lower bound on the minimax EER is provided. This bound is of the simple form $c_{\infty}/\sqrt{\nu}$ as $\nu\to\infty$, where $c_{\infty}=0.16997\dots$ is a universal constant, $\nu=m/d$, $m$ is the size of the training sample and $d$ is the Vapnik–Chervonenkis dimension of the hypothesis class. It is shown that the differences between these asymptotic and nonasymptotic bounds, as well as the differences between these two bounds and the maximum EER of any learning algorithms that minimize the empirical risk, are asymptotically negligible, and all these differences are due to ties in the mentioned “voting” procedures. A few easy to compute nonasymptotic lower bounds on the minimax EER are also obtained, which are shown to be close to the exact asymptotic lower bound $c_{\infty}/\sqrt{\nu}$ even for rather small values of the ratio $\nu=m/d$. As an application of these results, we substantially improve existing lower bounds on the tail probability of the excess risk. Among the tools used are Bayes estimation and apparently new identities and inequalities for binomial distributions.

Optimality of SVM: Novel proofs and tighter bounds

We provide a new proof that the expected error rate of consistent support vector machines matches the minimax rate (up to a constant factor) in its dependence on the sample size and margin. The upper bound was originally established by [1], while the lower bound follows from an argument of [2] together with reasoning about the VC dimension of large-margin classifiers. Our proof differs from the original in that many of our steps concern reasoning about the primal space, while the original carried out these steps by reasoning about the dual space. Our approach provides a unified framework for analyzing both the homogeneous and non-homogeneous cases, with slightly better results for the former. The fact that our analysis explicitly handles the non-homogeneous case offers significant improvements in the bounds compared to the usual textbook approach of reducing to the homogeneous case. We also extend our proof to provide a new upper bound on the error rate of transductive SVM, which yields an improved constant factor compared to inductive SVM. In addition to these bounds on the expected error rate, we also provide a simple proof of a margin-based PAC-style bound for support vector machines, and an extension of the agnostic PAC analysis that explicitly handles the non-homogeneous case.

ProSecCo: progressive sequence mining with convergence guarantees

We present PROSECCO, an algorithm for the progressive mining of frequent sequences from large transactional datasets: it processes the dataset in blocks and outputs, after having analyzed each block, a high-quality approximation of the collection of frequent sequences. These intermediate results have strong probabilistic approximation guarantees and the final output is the exact collection of frequent sequences. Our correctness analysis uses the Vapnik-Chervonenkis (VC) dimension, a key concept from statistical learning theory. The results of our experimental evaluation of PROSECCO on real and artificial datasets show that it produces fast-converging high-quality results almost immediately. Its practical performance is even better than what is guaranteed by the theoretical analysis, and it can even be faster than existing state-of-the-art non-progressive algorithms.

VC Dimension and a Union Theorem for Set Systems

Fix positive integers $k$ and $d$. We show that, as $n\to\infty$, any set system $\mathcal{A} \subset 2^{[n]}$ for which the VC dimension of $\{ \triangle_{i=1}^k S_i \mid S_i \in \mathcal{A}\}$ is at most $d$ has size at most $(2^{d\bmod{k}}+o(1))\binom{n}{\lfloor d/k\rfloor}$. Here $\triangle$ denotes the symmetric difference operator. This is a $k$-fold generalisation of a result of Dvir and Moran, and it settles one of their questions. 
 A key insight is that, by a compression method, the problem is equivalent to an extremal set theoretic problem on $k$-wise intersection or union that was originally due to Erdős and Frankl.
 We also give an example of a family $\mathcal{A} \subset 2^{[n]}$ such that the VC dimension of $\mathcal{A}\cap \mathcal{A}$ and of $\mathcal{A}\cup \mathcal{A}$ are both at most $d$, while $\lvert \mathcal{A} \rvert = \Omega(n^d)$. This provides a negative answer to another question of Dvir and Moran.

Structural Risk Minimization-Driven Genetic Programming for Enhancing Generalization in Symbolic Regression

Generalization ability, which reflects the prediction ability of a learned model, is an important property in genetic programming (GP) for symbolic regression. Structural risk minimization (SRM) is a framework providing a reliable estimation of the generalization performance of prediction models. Introducing the framework into GP has the potential to drive the evolutionary process toward models with good generalization performance. However, this is tough due to the difficulty in obtaining the Vapnik–Chervonenkis (VC) dimension of nonlinear models. To address this difficulty, this paper proposes an SRM-driven GP approach, which uses an experimental method (instead of theoretical estimation) to measure the VC dimension of a mixture of linear and nonlinear regression models for the first time. The experimental method has been conducted using uniform and nonuniform settings. The results show that our method has impressive generalization gains over standard GP and GP with the 0.632 bootstrap, and that the proposed method using the nonuniform setting has further improvement than its counterpart using the uniform setting. Further analyzes reveal that the proposed method can evolve more compact models, and that the behavioral difference between these compact models and the target models is much smaller than their counterparts evolved by the other GP methods.

Near-optimal Linear Decision Trees for k-SUM and Related Problems

We construct near-optimal linear decision trees for a variety of decision problems in combinatorics and discrete geometry. For example, for any constant k , we construct linear decision trees that solve the k -SUM problem on n elements using O ( n log 2 n ) linear queries. Moreover, the queries we use are comparison queries, which compare the sums of two k -subsets; when viewed as linear queries, comparison queries are 2 k -sparse and have only { −1,0,1} coefficients. We give similar constructions for sorting sumsets A+B and for solving the SUBSET-SUM problem, both with optimal number of queries, up to poly-logarithmic terms. Our constructions are based on the notion of “inference dimension,” recently introduced by the authors in the context of active classification with comparison queries. This can be viewed as another contribution to the fruitful link between machine learning and discrete geometry, which goes back to the discovery of the VC dimension.

Transit sets of [formula omitted]-point crossover operators

k-point crossover operators and their recombination sets are studied from different perspectives. We show that transit functions of k-point crossover generate, for all k>1, the same convexity as the interval function of the underlying graph. This settles in the negative an open problem by Mulder about whether the geodesic convexity of a connected graph G is uniquely determined by its interval function I. The conjecture of Gitchoff and Wagner that for each transit set Rk(x,y) distinct from a hypercube there is a unique pair of parents from which it is generated is settled affirmatively. Along the way we characterize transit functions whose underlying graphs are Hamming graphs, and those with underlying partial cube graphs. For general values of k it is shown that the transit sets of k-point crossover operators are the subsets with maximal Vapnik–Chervonenkis dimension.

The Exact VC Dimension of the WiSARD -Tuple Classifier.

The Wilkie, Stonham, and Aleksander recognition device (WiSARD) -tuple classifier is a multiclass weightless neural network capable of learning a given pattern in a single step. Its architecture is determined by the number of classes it should discriminate. A target class is represented by a structure called a discriminator, which is composed of RAM nodes, each of them addressed by an -tuple. Previous studies were carried out in order to mitigate an important problem of the WiSARD -tuple classifier: having its RAM nodes saturated when trained by a large data set. Finding the VC dimension of the WiSARD -tuple classifier was one of those studies. Although no exact value was found, tight bounds were discovered. Later, the bleaching technique was proposed as a means to avoid saturation. Recent empirical results with the bleaching extension showed that the WiSARD -tuple classifier can achieve high accuracies with low variance in a great range of tasks. Theoretical studies had not been conducted with that extension previously. This work presents the exact VC dimension of the basic two-class WiSARD -tuple classifier, which is linearly proportional to the number of RAM nodes belonging to a discriminator, and exponentially to their addressing tuple length, precisely . The exact VC dimension of the bleaching extension to the WiSARD -tuple classifier, whose value is the same as that of the basic model, is also produced. Such a result confirms that the bleaching technique is indeed an enhancement to the basic WiSARD -tuple classifier as it does no harm to the generalization capability of the original paradigm.

Erdős–Hajnal Conjecture for Graphs with Bounded VC-Dimension

The Vapnik–Chervonenkis dimension (in short, VC-dimension) of a graph is defined as the VC-dimension of the set system induced by the neighborhoods of its vertices. We show that every n-vertex graph with bounded VC-dimension contains a clique or an independent set of size at least $$e^{(\log n)^{1 - o(1)}}$$ . The dependence on the VC-dimension is hidden in the o(1) term. This improves the general lower bound, $$e^{c\sqrt{\log n}}$$ , due to Erdős and Hajnal, which is valid in the class of graphs satisfying any fixed nontrivial hereditary property. Our result is almost optimal and nearly matches the celebrated Erdős–Hajnal conjecture, according to which one can always find a clique or an independent set of size at least $$e^{\Omega (\log n)}$$ . Our results partially explain why most geometric intersection graphs arising in discrete and computational geometry have exceptionally favorable Ramsey-type properties. Our main tool is a partitioning result found by Lovasz–Szegedy and Alon–Fischer–Newman, which is called the “ultra-strong regularity lemma” for graphs with bounded VC-dimension. We extend this lemma to k-uniform hypergraphs, and prove that the number of parts in the partition can be taken to be $$(1/\varepsilon )^{O(d)}$$ , improving the original bound of $$(1/\varepsilon )^{O(d^2)}$$ in the graph setting. We show that this bound is tight up to an absolute constant factor in the exponent. Moreover, we give an $$O(n^k)$$ -time algorithm for finding a partition meeting the requirements. Finally, we establish tight bounds on Ramsey–Turan numbers for graphs with bounded VC-dimension.

A Sauer-Shelah-Perles Lemma for Sumsets

We show that any family of subsets $A\subseteq 2^{[n]}$ satisfies $\lvert A\rvert \leq O\bigl(n^{\lceil{d}/{2}\rceil}\bigr)$, where $d$ is the VC dimension of $\{S\triangle T \,\vert\, S,T\in A\}$, and $\triangle$ is the symmetric difference operator. We also observe that replacing $\triangle$ by either $\cup$ or $\cap$ fails to satisfy an analogous statement. Our proof is based on the polynomial method; specifically, on an argument due to [Croot, Lev, Pach '17].

Optimal quantum sample complexity of learning algorithms

In learning theory, the VC dimension of a concept class C is the most common way to measure its “richness.” A fundamental result says that the number of examples needed to learn an unknown target concept c∈C under an unknown distribution D, is tightly determined by the VC dimension d of the concept class C. Specifically, in the PAC model Θ(dϵ+log(1/δ)ϵ) examples are necessary and sufficient for a learner to output, with probability 1−δ, a hypothesis h that is ϵ-close to the target concept c (measured under D). In the related agnostic model, where the samples need not come from a c∈C, we know that Θ(dϵ2+log(1/δ)ϵ2) examples are necessary and sufficient to output an hypothesis h∈C whose error is at most ϵ worse than the error of the best concept in C. Here we analyze quantum sample complexity, where each example is a coherent quantum state. This model was introduced by Bshouty and Jackson (1999), who showed that quantum examples are more powerful than classical examples in some fixed-distribution settings. However, Atici and Servedio (2005), improved by Zhang (2010), showed that in the PAC setting (where the learner has to succeed for every distribution), quantum examples cannot be much more powerful: the required number of quantum examples is Ω(d1−ηϵ+d+log(1/δ)ϵ) for arbitrarily small constant η>0. Our main result is that quantum and classical sample complexity are in fact equal up to constant factors in both the PAC and agnostic models. We give two proof approaches. The first is a fairly simple information-theoretic argument that yields the above two classical bounds and yields the same bounds for quantum sample complexity up to a log(d/ϵ) factor. We then give a second approach that avoids the log-factor loss, based on analyzing the behavior of the “Pretty Good Measurement” on the quantum state-identification problems that correspond to learning. This shows classical and quantum sample complexity are equal up to constant factors for every concept class C.

A Relational Framework for Classifier Engineering

In the design of analytical procedures and machine learning solutions, a critical and time-consuming task is that of feature engineering, for which various recipes and tooling approaches have been developed. In this article, we embark on the establishment of database foundations for feature engineering. We propose a formal framework for classification in the context of a relational database. The goal of this framework is to open the way to research and techniques to assist developers with the task of feature engineering by utilizing the database’s modeling and understanding of data and queries and by deploying the well-studied principles of database management. As a first step, we demonstrate the usefulness of this framework by formally defining three key algorithmic challenges. The first challenge is that of separability, which is the problem of determining the existence of feature queries that agree with the training examples. The second is that of evaluating the VC dimension of the model class with respect to a given sequence of feature queries. The third challenge is identifiability, which is the task of testing for a property of independence among features that are represented as database queries. We give preliminary results on these challenges for the case where features are defined by means of conjunctive queries, and, in particular, we study the implication of various traditional syntactic restrictions on the inherent computational complexity.

Parameterized and approximation complexity of Partial VC Dimension

We introduce the problem Partial VC Dimension that asks, given a hypergraph H=(X,E) and integers k and ℓ, whether one can select a set C⊆X of k vertices of H such that the set {e∩C,e∈E} of distinct hyperedge-intersections with C has size at least ℓ. The sets e∩C define equivalence classes over E. Partial VC Dimension is a generalization of VC Dimension, which corresponds to the case ℓ=2k, and of Distinguishing Transversal, which corresponds to the case ℓ=|E| (the latter is also known as Test Cover in the dual hypergraph). We also introduce the associated fixed-cardinality maximization problem Max Partial VC Dimension that aims at maximizing the number of equivalence classes induced by a solution set of k vertices. We study the algorithmic complexity of Partial VC Dimension and Max Partial VC Dimension both on general hypergraphs and on more restricted instances, in particular, neighborhood hypergraphs of graphs.

The Vapnik–Chervonenkis dimension of graph and recursive neural networks

The Vapnik–Chervonenkis dimension (VC-dim) characterizes the sample learning complexity of a classification model and it is often used as an indicator for the generalization capability of a learning method. The VC-dim has been studied on common feed-forward neural networks, but it has yet to be studied on Graph Neural Networks (GNNs) and Recursive Neural Networks (RecNNs). This paper provides upper bounds on the order of growth of the VC-dim of GNNs and RecNNs. GNNs and RecNNs are from a new class of neural network models which are capable of processing inputs that are given as graphs. A graph is a data structure that generalizes the representational power of vectors and sequences, via the ability to represent dependencies or relationships between feature vectors. It was shown previously that the ability of recurrent neural networks to process sequences increases the VC-dim when compared to the VC-dim of Neural Networks, which are limited to processing vectors. Since graphs are a more general form than sequences, the question arises how this will affect the VC-dimension of GNNs and RecNNs. A main finding in this paper is that the upper bounds on the VC-dim for GNNs and RecNNs are comparable to the upper bounds for recurrent neural networks. The result also suggests that the generalization capability of such models increases with the number of connected nodes.

Power quality assessment method of wind power grid connection based on least squares support vector machine

In this paper, according to the ‘Technical rule for connecting wind farm to power system’ promulgated by the state, the least squares support vector machine (LSSVM) algorithm is used to build a mathematical model, and six indicators, namely, voltage deviation, voltage fluctuation, voltage flicker, grid harmonics, frequency deviation and three-phase voltage imbalance, are comprehensively considered and a certain quality level is calculated, which is used to evaluate the power quality of wind power stations and substation access points. LSSVM is a classification method based on the principle of minimizing institutional risk and combined VC dimension theory. Compared with traditional classification method, LSSVM has the advantages of low complexity, short training time, high accuracy and the like. According to the electric energy data provided by Weifang power grid, the accuracy and practicability of this evaluation method are verified.

On density of subgraphs of halved cubes

Let S be a family of subsets of a set X of cardinality m and VC-dim(S) be the Vapnik–Chervonenkis dimension of S. Haussler et al. (1994) proved that if G1(S)=(V,E) is the subgraph of the hypercube Qm induced by S (called the 1-inclusion graph of S), then |E||V|≤VC-dim(S). Haussler (1995) presented an elegant proof of this inequality using the shifting operation.In this note, we adapt the shifting technique to prove that if S is an arbitrary set family and G1,2(S)=(V,E) is the 1,2-inclusion graph of S (i.e., the subgraph of the square Qm2 of the hypercube Qm induced by S), then |E||V|≤d2, where d≔cVC-dim∗(S) is the clique-VC-dimension of S (which we introduce in this paper). The 1,2-inclusion graphs are exactly the subgraphs of halved cubes and comprise subgraphs of Johnson graphs as a subclass.

Complexity of concept classes induced by discrete Markov networks and Bayesian networks

Markov networks and Bayesian networks are two popular models for classification. Vapnik–Chervonenkis dimension and Euclidean dimension are two measures of complexity of a class of functions, which can be used to measure classification capability of classifiers. One can use Vapnik–Chervonenkis dimension of the class of functions associated with a classifier to construct an estimate of its generalization error. In this paper, we study Vapnik–Chervonenkis dimension and Euclidean dimension of concept classes induced by discrete Markov networks and Bayesian networks. We show that these two dimensional values of the concept class induced by a discrete Markov network are identical, and the value equals dimension of the toric ideal corresponding to this Markov network as long as the toric ideal is nontrivial. Based on this result, one can compute the dimensional value in terms of a computer algebra system directly. Furthermore, for a general Bayesian network, we show that dimension of the corresponding toric ideal offers an upper bound of Euclidean dimension. In addition, we illustrate how to use Vapnik–Chervonenkis dimension to estimate generalization error in binary classification.

Two results about the hypercube

First we consider families in the hypercube Qn with bounded VC dimension. Frankl raised the problem of estimating the number m(n,k) of maximum families of VC dimension k. Alon, Moran and Yehudayoff showed that n(1+o(1))1k+1nk≤m(n,k)≤n(1+o(1))nk.We close the gap by showing that logm(n,k)=(1+o(1))nklogn. We also show how a bound on the number of induced matchings between two adjacent small layers of Qn follows as a corollary.Next, we consider the integrity I(Qn) of the hypercube, defined as I(Qn)=min{|S|+m(Qn∖S):S⊆V(Qn)},where m(H) denotes the number of vertices in the largest connected component of H. Beineke, Goddard, Hamburger, Kleitman, Lipman and Pippert showed that c2nn≤I(Qn)≤C2nnlogn and suspected that their upper bound is the right value. We prove that the truth lies below the upper bound by showing that I(Qn)≤C2nnlogn.

СТАТИСТИЧЕСКАЯ ЗНАЧИМОСТЬ ПРОГНОЗИРОВАНИЯ РЕЗУЛЬТАТОВ ПРОИЗВОДСТВЕННОГО ПРОЦЕССА С ПОМОЩЬЮ ИСКУССТВЕННОЙ НЕЙРОННОЙ СЕТИ

As a result of the research of production process organization for the roof construction of residential multi-storey buildings, an artificial neural network (ANN) was designed, the purpose of which is to predict the labor productivity based on organizational factors. One of the main tasks on the way to this purpose is the training of ANN on precedents of the sample extracted from the research object. In view of the deficiency of training data, the main problem is to determine the conditions for the statistical significance of the predictions of the model trained on limited sample. This article is devoted to solving this problem within the research of production organization. The paper uses the provisions of the statistical learning theory, the notion of the Vapnik-Chervonenkis dimension for describing the sample complexity, and also the approaches of probably approximately correct learning (PAC-learning). The technologies of statistical bootstrapping and bagging are described, which allow expanding the training sample. ANN training is conducted using a computer experiment on the programming language Python. The bounds of the theoretical sample complexity, which is necessary for obtaining of ANN results within a given confidence interval with a confidence level of 0,95, were estimated. The sample was transformed by an order comparable to the theoretical lower bound. ANN was trained and the mean square error (MSE) in the test sample was defined, which amounted to . The theoretical bounds of the sample complexity to ensure a given statistical significance are determined in the article. After the ANN training on the sample, the order of which corresponds to theoretical lower bound, a prediction error was obtained on the test sample within the given confidence interval.