VC-dimension Research Articles

Theory and algorithm for domain adaptation

Domain adaptation is an important subfield of transfer learning, it has been successfully applied in many applications of machine learning. Recently, significant theoretical and algorithmic advances have been achieved in domain adaptation. The theoretical analyses for domain adaptation are based on VC dimension and Rademacher complexity. There are also some covering number-based results, but most of these bounds are based on the results of Rademacher complexity, indirectly given by the relationship between covering number and Rademacher complexity. In this paper, we propose a theoretical analysis framework for domain adaptation, thus the error bound can be derived directly by covering number, which is an effective method for analyzing the generalization error in statistical learning theory. We derive generalization error bound for domain adaptation with a class of loss functions satisfying the assumptions. We also propose a mixup contrastive adversarial network for domain adaptation by introducing a mixup module for enhancing the alignment of the source and target domains during domain transfer, and a contrastive learning module for solving class-level alignment after domain transfer. Experimental results demonstrate the effectiveness of the proposed algorithm and the property of the theoretical results.

Classifier-dependent feature selection via greedy methods

The purpose of this study is to introduce a new approach to feature ranking for classification tasks, called in what follows greedy feature selection. In statistical learning, feature selection is usually realized by means of methods that are independent of the classifier applied to perform the prediction using that reduced number of features. Instead, the greedy feature selection identifies the most important feature at each step and according to the selected classifier. The benefits of such scheme are investigated in terms of model capacity indicators, such as the Vapnik-Chervonenkis dimension or the kernel alignment. This theoretical study proves that the iterative greedy algorithm is able to construct classifiers whose complexity capacity grows at each step. The proposed method is then tested numerically on various datasets and compared to the state-of-the-art techniques. The results show that our iterative scheme is able to truly capture only a few relevant features, and may improve, especially for real and noisy data, the accuracy scores of other techniques. The greedy scheme is also applied to the challenging application of predicting geo-effective manifestations of the active Sun.

A class-aware representation refinement framework for graph classification

Graph Neural Networks (GNNs) are widely used for graph representation learning. Despite its prevalence, GNN suffers from two drawbacks in the graph classification task, the neglect of graph-level relationships, and the generalization issue. Each graph is treated separately in GNN message passing/graph pooling, and existing methods to address overfitting operate on each individual graph. This makes the graph representations learnt less effective in the downstream classification. In this paper, we propose a Class-Aware Representation rEfinement (CARE) framework for the task of graph classification. CARE computes simple yet powerful class representations and injects them to steer the learning of graph representations towards better class separability. CARE is a plug-and-play framework that is highly flexible and able to incorporate arbitrary GNN backbones without significantly increasing the computational cost. We also theoretically prove that CARE has a better generalization upper bound than its GNN backbone through Vapnik-Chervonenkis (VC) dimension analysis. Our extensive experiments with 11 well-known GNN backbones on 9 benchmark datasets validate the superiority and effectiveness of CARE over its GNN counterparts.

Smoothed Analysis with Adaptive Adversaries

We prove novel algorithmic guarantees for several online problems in the smoothed analysis model. In this model, at each time step an adversary chooses an input distribution with density function bounded above pointwise by \(\tfrac{1}{\sigma }\) times that of the uniform distribution; nature then samples an input from this distribution. Here, σ is a parameter that interpolates between the extremes of worst-case and average case analysis. Crucially, our results hold for adaptive adversaries that can base their choice of input distribution on the decisions of the algorithm and the realizations of the inputs in the previous time steps. An adaptive adversary can nontrivially correlate inputs at different time steps with each other and with the algorithm’s current state; this appears to rule out the standard proof approaches in smoothed analysis. This paper presents a general technique for proving smoothed algorithmic guarantees against adaptive adversaries, in effect reducing the setting of an adaptive adversary to the much simpler case of an oblivious adversary (i.e., an adversary that commits in advance to the entire sequence of input distributions). We apply this technique to prove strong smoothed guarantees for three different problems: (1) Online learning: We consider the online prediction problem, where instances are generated from an adaptive sequence of σ-smooth distributions and the hypothesis class has VC dimension d . We bound the regret by \(\tilde{O}(\sqrt {T d\ln (1/\sigma)} + d\ln (T/\sigma))\) and provide a near-matching lower bound. Our result shows that under smoothed analysis, learnability against adaptive adversaries is characterized by the finiteness of the VC dimension. This is as opposed to the worst-case analysis, where online learnability is characterized by Littlestone dimension (which is infinite even in the extremely restricted case of one-dimensional threshold functions). Our results fully answer an open question of Rakhlin et al. [ 64 ]. (2) Online discrepancy minimization: We consider the setting of the online Komlós problem, where the input is generated from an adaptive sequence of σ-smooth and isotropic distributions on the ℓ 2 unit ball. We bound the ℓ ∞ norm of the discrepancy vector by \(\tilde{O}(\ln ^2(\frac{nT}{\sigma }))\) . This is as opposed to the worst-case analysis, where the tight discrepancy bound is \(\Theta (\sqrt {T/n})\) . We show such \(\mathrm{polylog}(nT/\sigma)\) discrepancy guarantees are not achievable for non-isotropic σ-smooth distributions. (3) Dispersion in online optimization: We consider online optimization with piecewise Lipschitz functions where functions with ℓ discontinuities are chosen by a smoothed adaptive adversary and show that the resulting sequence is \(({\sigma }/{\sqrt {T\ell }}, \tilde{O}(\sqrt {T\ell }))\) -dispersed. That is, every ball of radius \({\sigma }/{\sqrt {T\ell }}\) is split by \(\tilde{O}(\sqrt {T\ell })\) of the partitions made by these functions. This result matches the dispersion parameters of Balcan et al. [ 13 ] for oblivious smooth adversaries, up to logarithmic factors. On the other hand, worst-case sequences are trivially (0, T )-dispersed. 1

Learning Decision-Making Functions Given Cardinal and Ordinal Consensus Data

Decision-making and reaching consensus are an integral part of everyday life, and studying how individuals reach these decisions is an important problem in psychology, economics, and social choice theory. Our work develops methods and theory for learning the nature of decisions reached upon by individual decision makers or groups of individuals using data. We consider two tasks, where we have access to data on: 1) Cardinal utilities for d individuals with cardinal consensus values that the group or decision maker arrives at, 2) Cardinal utilities for d individuals for pairs of actions, with ordinal information about the consensus, i.e., which action is better according to the consensus. Under some axioms of social choice theory, the set of possible decision functions reduces to the set of weighted power means, M(u, w, p) = (∑ᵢ₌₁ᵈ wᵢ uᵢᵖ)¹ᐟᵖ, where uᵢ indicate the d utilities, w ∈ ∆_{d - 1} denotes the weights assigned to the d individuals, and p ∈ ℝ (Cousins 2023). For instance, p = 1 corresponds to a weighted utilitiarian function, and p = -∞ is the egalitarian welfare function. Our goal is to learn w ∈ ∆_{d - 1} and p ∈ ℝ for the two tasks given data. The first task is analogous to regression, and we show that owing to the monotonicity in w and p (Qi 2000}, learning these parameters given cardinal utilities and social welfare values is a PAC-learnable task. For the second task, we wish to learn w, p such that, given pairs of actions u, v ∈ ℝ₊ᵈ, the preference is given as C((u, v), w, p) = sign(ln(M(u, w, p)) - ln(M(v, w, p))). This is analogous to classification; however, convexity of the loss function in w and p is not guaranteed. We analyze two related cases - one in which the weights w are known, and another in which the weights are unknown. We prove that both cases are PAC-learnable given positive u, v by giving an O(log d) bound on the VC dimension for the known weights case, and an O(d log d) bound for the unknown weights case. We also establish PAC-learnability for noisy data under IID (Natarajan 2013) and logistic noise models for this task. Finally, we demonstrate how simple algorithms can be useful to learn w and p up to moderately high d through experiments on simulated data.

Attention non-negative spectral clustering

Deep spectral clustering is an advanced unsupervised deep learning method with widespread applications. However, when applied to large-scale datasets, it may confront problems such as an unstable training process, limited scalability, and diminished interpretability. To mitigate these limitations, a novel unsupervised deep learning algorithm named Attention Non-negative Spectral Clustering (ANSC) is proposed. Specifically, a network architecture called the Spectral Attention Network (SAN) is designed to serve as the backbone network for ANSC. This design aims to improve the stability of the training process and the model’s scalability. This attention-based network architecture includes a feature extraction network, an eigenvector mapping network, and an orthogonalization module. Furthermore, to enhance the interpretability of ANSC, the objective function incorporates a cluster indicator matrix by introducing non-negative constraints. Additionally, the ℓ2,1-norm is employed as regularization to approximate the solution of the objective function. An analysis of a lower bound for the network size of ANSC is conducted based on the VC dimension. Extensive experiments validate that ANSC outperforms state-of-the-art clustering methods in terms of clustering accuracy, normalized mutual information, and adjusted rand index.

Unlabeled sample compression schemes for oriented matroids

A long-standing sample compression conjecture asks to linearly bound the size of the optimal sample compression schemes by the Vapnik-Chervonenkis (VC) dimension of an arbitrary class. In this paper, we explore the rich metric and combinatorial structure of oriented matroids (OMs) to construct proper unlabeled sample compression schemes for the classes of topes of OMs bounded by their VC-dimension. The result extends to the topes of affine OMs, as well as to the topes of the complexes of OMs that possess a corner peeling. The main tool that we use are the solutions of certain oriented matroid programs.

Understanding quantum machine learning also requires rethinking generalization

Quantum machine learning models have shown successful generalization performance even when trained with few data. In this work, through systematic randomization experiments, we show that traditional approaches to understanding generalization fail to explain the behavior of such quantum models. Our experiments reveal that state-of-the-art quantum neural networks accurately fit random states and random labeling of training data. This ability to memorize random data defies current notions of small generalization error, problematizing approaches that build on complexity measures such as the VC dimension, the Rademacher complexity, and all their uniform relatives. We complement our empirical results with a theoretical construction showing that quantum neural networks can fit arbitrary labels to quantum states, hinting at their memorization ability. Our results do not preclude the possibility of good generalization with few training data but rather rule out any possible guarantees based only on the properties of the model family. These findings expose a fundamental challenge in the conventional understanding of generalization in quantum machine learning and highlight the need for a paradigm shift in the study of quantum models for machine learning tasks.

Parallelograms and the VC-dimension of the distance sets

In this paper, we study the distribution of parallelograms and rhombi in a given set in the plane over arbitrary finite fields Fq2. As an application, we improve the recent result due to Fitzpatrick, Iosevich, McDonald, and Wyman on the Vapnik–Chervonenkis dimension of the induced distance graph.

VC dimensions of group convolutional neural networks

We study the generalization capacity of group convolutional neural networks. We identify precise estimates for the VC dimensions of simple sets of group convolutional neural networks. In particular, we find that for infinite groups and appropriately chosen convolutional kernels, already two-parameter families of convolutional neural networks have an infinite VC dimension, despite being invariant to the action of an infinite group.

Compression schemes for concept classes induced by three types of discrete undirected graphical models

Sample compression schemes were first proposed by Littlestone and Warmuth in 1986. Undirected graphical model is a powerful tool for classification in statistical learning. In this paper, we consider labelled compression schemes for concept classes induced by discrete undirected graphical models. For the undirected graph of two vertices with no edge, where one vertex takes two values and the other vertex can take any finite number of values, we propose an algorithm to establish a labelled compression scheme of size VC dimension of associated concept class. Further, we extend the result to other two types of undirected graphical models and show the existence of labelled compression schemes of size VC dimension for induced concept classes. The work of this paper makes a step forward in solving sample compression problem for concept class induced by a general discrete undirected graphical model.

A Range Space with Constant VC Dimension for All-pairs Shortest Paths in Graphs

Let $G$ be an undirected graph with non-negative edge weights and let $S$ be a subset of its shortest paths such that, for every pair $(u,v)$ of distinct vertices, $S$ contains exactly one shortest path between $u$ and $v$. In this paper we define a range space associated with $S$ and prove that its VC dimension is 2. As a consequence, we show a bound for the number of shortest paths trees required to be sampled in order to solve a relaxed version of the All-pairs Shortest Paths problem (APSP) in $G$. In this version of the problem we are interested in computing all shortest paths with a certain "importance" at least $\varepsilon$. Given any $0 < \varepsilon$, $ \delta < 1$, we propose a $\mathcal{O}(m + n \log n + (\textrm{diam}_{V(G)})^2)$ sampling algorithm that outputs with probability $1 - \delta$ the (exact) distance and the shortest path between every pair of vertices $(u, v)$ that appears as subpath of at least a proportion $\varepsilon$ of all shortest paths in the set $S$, where $\textrm{diam}_{V(G)}$ is the vertex-diameter of $G$. The bound that we obtain for the sample size depends only on $\varepsilon$ and $\delta$, and do not depend on the size of the graph.

Boosting Simple Learners

Boosting is a celebrated machine learning approach which is based on the idea of combining weak and moderately inaccurate hypotheses to a strong and accurate one. We study boosting under the assumption that the weak hypotheses belong to a class of bounded capacity. This assumption is inspired by the common convention that weak hypotheses are "rules-of-thumbs" from an "easy-to-learn class". (Schapire and Freund~'12, Shalev-Shwartz and Ben-David '14.) Formally, we assume the class of weak hypotheses has a bounded VC dimension. We focus on two main questions: (i) Oracle Complexity: How many weak hypotheses are needed to produce an accurate hypothesis? We design a novel boosting algorithm and demonstrate that it circumvents a classical lower bound by Freund and Schapire ('95, '12). Whereas the lower bound shows that $\Omega({1}/{\gamma^2})$ weak hypotheses with $\gamma$-margin are sometimes necessary, our new method requires only $\tilde{O}({1}/{\gamma})$ weak hypothesis, provided that they belong to a class of bounded VC dimension. Unlike previous boosting algorithms which aggregate the weak hypotheses by majority votes, the new boosting algorithm uses more complex ("deeper") aggregation rules. We complement this result by showing that complex aggregation rules are in fact necessary to circumvent the aforementioned lower bound. (ii) Expressivity: Which tasks can be learned by boosting weak hypotheses from a bounded VC class? Can complex concepts that are "far away" from the class be learned? Towards answering the first question we {introduce combinatorial-geometric parameters which capture expressivity in boosting.} As a corollary we provide an affirmative answer to the second question for well-studied classes, including half-spaces and decision stumps. Along the way, we establish and exploit connections with Discrepancy Theory.

Lossy Compression With Universal Distortion

We consider a novel variant of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">d</i> -semifaithful lossy coding in which the distortion measure is revealed only to the encoder and only at run-time, as well as an extension of it in which the distortion constraint <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">d</i> is also revealed at run-time. Two forms of rate redundancy are used to analyze the performance, and achievability results of both a pointwise and minimax nature are demonstrated. The first coding scheme uses ideas from VC dimension and growth functions, the second uses appropriate quantization of the space of distortion measures, and the third relies on a random coding argument.

An Optimal Transport Analysis on Generalization in Deep Learning.

Deep neural networks (DNNs) have achieved state-of-the-art performance in various learning tasks, such as computer vision, natural language processing, and speech recognition. However, the fundamental theory of generalization still remains obscure in deep learning-why DNN models can generalize well, despite that they are heavily overparametrized in both depth and width? Recently, some work shows that traditional theory of analyzing the generalization error of learning models fails to explain the generalization of DNNs. The failure is mainly because of one simple fact that the worse case analysis of generalization error for learning models would be too loose for models with large parameter space, such as DNNs. In this work, we propose a new analysis of generalization in DNNs from an optimal transport perspective. Unlike traditional worse-case uniform convergence analysis in learning theory, our analysis of generalization error is dependent on both the learning algorithm and the data distribution and is the average-case analysis. Thus, our theory can be more practical and accurate to describe the generalization behavior of DNNs. More specifically, in this article, we try to answer a fundamental yet unsolved question in deep learning-why deeper models can generalize well than shallow models? The main contribution of this article can be summarized in four aspects. First, under a general learning framework, we derive upper bounds on the generalization error of learning algorithms by their algorithmic transport cost: the expected Wasserstein distance between the output hypothesis and the output hypothesis conditioned on an input example. We further provide several upper bounds on the algorithmic transport cost in terms of total variation distance, relative entropy, and Vapnik-Chervonenkis (VC) dimension. Moreover, we also study different conditions for loss functions under which the generalization error of a learning algorithm can be upper bounded by different probability metrics between distributions relating to the output hypothesis and/or the input data. Finally, under our established framework, we obtain our main results, showing that the generalization error in DNNs decreases exponentially to zero as the number of layers increases.

Induced odd cycle packing number, independent sets, and chromatic number

AbstractThe induced odd cycle packing number of a graph is the maximum integer such that contains an induced subgraph consisting of pairwise vertex‐disjoint odd cycles. Motivated by applications to geometric graphs, Bonamy et al. proved that graphs of bounded induced odd cycle packing number, bounded Vapnik–Chervonenkis (VC) dimension, and linear independence number admit a randomized efficient polynomial‐time approximation scheme for the independence number. We show that the assumption of bounded VC dimension is not necessary, exhibiting a randomized algorithm that for any integers and and any ‐vertex graph of induced odd cycle packing number at most returns in time an independent set of whose size is at least with high probability. In addition, we present ‐boundedness results for graphs with bounded odd cycle packing number, and use them to design a quasipolynomial‐time approximation scheme for the independence number only assuming bounded induced odd cycle packing number.

Minimum Excess Risk in Bayesian Learning

We analyze the best achievable performance of Bayesian learning under generative models by defining and upper-bounding the minimum excess risk (MER): the gap between the minimum expected loss attainable by learning from data and the minimum expected loss that could be achieved if the model realization were known. The definition of MER provides a principled way to define different notions of uncertainties in Bayesian learning, including the aleatoric uncertainty and the minimum epistemic uncertainty. Two methods for deriving upper bounds for the MER are presented. The first method, generally suitable for Bayesian learning with a parametric generative model, upper-bounds the MER by the conditional mutual information between the model parameters and the quantity being predicted given the observed data. It allows us to quantify the rate at which the MER decays to zero as more data becomes available. Under realizable models, this method also relates the MER to the richness of the generative function class, notably the VC dimension in binary classification. The second method, particularly suitable for Bayesian learning with a parametric predictive model, relates the MER to the minimum estimation error of the model parameters from data. It explicitly shows how the uncertainty in model parameter estimation translates to the MER and to the final prediction uncertainty. We also extend the definition and analysis of MER to the setting with multiple model families and the setting with nonparametric models. Along the discussions we draw some comparisons between the MER in Bayesian learning and the excess risk in frequentist learning.

ENSIGHTS: Intelligent Monitoring of Electric Power Transmission Assets

This work aims to use Data Science techniques to build predictive models that will eventually improve maintenance plans regarding power transformers by reducing shutoffs and transmission downtime. This work is part of a 36-month long Research and Development (R&amp;D) project started on January 2021, as of the writing of this report the project is halfway through. The analytic models described herein have already been tested while most of the remaining work is yet to be done. In a single Data Lake environment, we will consolidate several databases from information systems that support the company's operation and maintenance processes. Various machine learning models will run on this data, and their results will appear in a dashboard, alongside several traditional indicators. This process will be integrated and consolidated in a computing platform with cloud architecture. We use Machine Learning (ML) to develop the models. Based on an Agnostic Probably Approximately Correct (PAC) Learning study of the available datasets estimating their Vapnik–Chervonenkis dimension, we choose Random Forests (RF) algorithms to be used in the new indicators. So far, the project has produced two new indicators: Chromatographic Assay Indicator (CAI) and Electrical Failure Risk Indicator (EFRI). The CAI indicator evaluation uses a Random Forest Algorithm trained with an external dataset due to the small number of power transformers failures in the O&amp;M data. This indicator performed much better than classical chromatographic indicators to predict electric or temperature problems on the test set. The EFRI indicator correlates monitoring data available from an existing Supervisory Control and Data Acquisition (SCADA) system with maintenance data from an existing Enterprise Resource Planning (ERP) system through a RF algorithm capable of alerting to a higher risk of electric failure. ANEEL funds this work as R&amp;D project PD-00394-1907/2019 titled “Aplicabilidade de nova tecnologia voltada para o desenvolvimento de um modelo de monitoramento inteligente dos ativos de transmissão”.

ENSIGHTS: Intelligent Monitoring of Electric Power Transmission Assets

This work aims to use Data Science techniques to build predictive models that will eventually improve maintenance plans regarding power transformers by reducing shutoffs and transmission downtime. This work is part of a 36-month long Research and Development (R&amp;D) project started on January 2021, as of the writing of this report the project is halfway through. The analytic models described herein have already been tested while most of the remaining work is yet to be done. In a single Data Lake environment, we will consolidate several databases from information systems that support the company's operation and maintenance processes. Various machine learning models will run on this data, and their results will appear in a dashboard, alongside several traditional indicators. This process will be integrated and consolidated in a computing platform with cloud architecture. We use Machine Learning (ML) to develop the models. Based on an Agnostic Probably Approximately Correct (PAC) Learning study of the available datasets estimating their Vapnik–Chervonenkis dimension, we choose Random Forests (RF) algorithms to be used in the new indicators. So far, the project has produced two new indicators: Chromatographic Assay Indicator (CAI) and Electrical Failure Risk Indicator (EFRI). The CAI indicator evaluation uses a Random Forest Algorithm trained with an external dataset due to the small number of power transformers failures in the O&amp;M data. This indicator performed much better than classical chromatographic indicators to predict electric or temperature problems on the test set. The EFRI indicator correlates monitoring data available from an existing Supervisory Control and Data Acquisition (SCADA) system with maintenance data from an existing Enterprise Resource Planning (ERP) system through a RF algorithm capable of alerting to a higher risk of electric failure. ANEEL funds this work as R&amp;D project PD-00394-1907/2019 titled “Aplicabilidade de nova tecnologia voltada para o desenvolvimento de um modelo de monitoramento inteligente dos ativos de transmissão”.

General Vapnik–Chervonenkis dimension bounds for quantum circuit learning

Quantifying the model complexity of quantum circuits provides a guide to avoid overfitting in quantum machine learning. Previously we established a Vapnik–Chervonenkis (VC) dimension upper bound for ‘encoding-first’ quantum circuits, where the input layer is the first layer of the circuit. In this work, we prove a general VC dimension upper bound for quantum circuit learning including ‘data re-uploading’ circuits, where the input gates can be single qubit rotations anywhere in the circuit. A linear lower bound is also constructed. The properties of the bounds and approximation-estimation trade-off considerations are discussed.