Vapnik-Chervonenkis Theory Research Articles

To understand double descent, we need to understand VC theory

We analyze generalization performance of over-parameterized learning methods for classification, under VC-theoretical framework. Recently, practitioners in Deep Learning discovered ‘double descent’ phenomenon, when large networks can fit perfectly available training data, and at the same time, achieve good generalization for future (test) data. The current consensus view is that VC-theoretical results cannot account for good generalization performance of Deep Learning networks. In contrast, this paper shows that double descent can be explained by VC-theoretical concepts, such as VC-dimension and Structural Risk Minimization. We also present empirical results showing that double descent generalization curves can be accurately modeled using classical VC-generalization bounds. Proposed VC-theoretical analysis enables better understanding of generalization curves for data sets with different statistical characteristics, such as low vs high-dimensional data and noisy data. In addition, we analyze generalization performance of transfer learning using pre-trained Deep Learning networks.

On Consistent Entropy-Regularized k-Means Clustering With Feature Weight Learning: Algorithm and Statistical Analyses.

Clusters in real data are often restricted to low-dimensional subspaces rather than the entire feature space. Recent approaches to circumvent this difficulty are often computationally inefficient and lack theoretical justification in terms of their large-sample behavior. This article deals with the problem by introducing an entropy incentive term to efficiently learn the feature importance within the framework of center-based clustering. A scalable block-coordinate descent algorithm, with closed-form updates, is incorporated to minimize the proposed objective function. We establish theoretical guarantees on our method by Vapnik-Chervonenkis (VC) theory to establish strong consistency along with uniform concentration bounds. The merits of our method are showcased through detailed experimental analysis on toy examples as well as real data clustering benchmarks.

Reinforced SVM method and memorization mechanisms

The paper is devoted to two problems: (1) reinforcement of SVM algorithms, and (2) justification of memorization mechanisms for generalization.(1) Current SVM algorithm was designed for the case when the risk for the set of nonnegative slack variables is defined by l1 norm. In this paper, along with that classical l1 norm, we consider risks defined by l2 norm and l∞ norm. Using these norms, we formulate several modifications of the existing SVM algorithm and show that the resulting modified SVM algorithms can improve (sometimes significantly) the classification performance.(2) Generalization ability of existing learning algorithms is usually explained by arguments involving uniform convergence of empirical losses to the corresponding expected losses over a given set of functions. However, along with bounds for uniform convergence of empirical losses to the expected losses, the VC theory also provides bounds for relative uniform convergence. These bounds lead to a more accurate estimate of the expected loss. Advanced methods of estimating of expected risk of error have to leverage these bounds, which also support mechanisms of training data memorization, which, as the paper demonstrates, can improve classification performance.

On uniform concentration bounds for Bi-clustering by using the Vapnik–Chervonenkis theory

Bi-clustering refers to the task of partitioning the rows and columns of a data matrix simultaneously. Although empirically useful, the theoretical aspects of bi-clustering techniques have not been studied in-depth. We present a framework for investigating the statistical guarantees behind the sparse bi-clustering algorithm by using the Vapnik–Chervonenkis (VC) theory.

Model Karmaşıklığının Kontrolü

Model karmaşıklığı, modellerin başarısındaki en önemli ölçütlerden birisidir. Bu çalışmada bugüne dek model karmaşıklığının kontrolünde öne çıkan yaklaşımlar başlıklar halinde incelenmiştir. Bunlar Occam’ın usturası, Popper’ın yanlışlanabilirliği ve istatiksel öğrenme teorisidir. Occam’ın usturası ve Popper’ın yanlışlanabilirliği model karmaşıklığının kontrolünde, evet bir felsefi yaklaşım sağlamaktadırlar ve kabul de görmektedirler. Fakat model karmaşıklığının nasıl kontrol edileceği konusunda matematiksel bir formülasyon sağlamamaktadırlar. Fakat istatiksel öğrenme teorisi (diğer adıyla, VC teorisi) konuya yaklaşımı yalnızca felsefi bir düzeyde kalmamakta, aynı zamanda şimdiye dek geliştirilen modellerde kullanılan ampirik risk minimizasyonu (ARM) ilkesine VC katsayısını ilave ederek yeni bir risk minimizasyonu (yapısal risk minimizasyonu, YRM) ilkesi getirmektedir. Sonuç olarak Vapnik ve Chervonenkis tarafından geliştirilen VC teorisi bir kontrol modeli olarak, ispatlanmış matematiksel arka planı ve oldukça başarılı olan sonuçları itibariyle, model karmaşıklığının kontrolü konusunda, günümüz çerçevesinde, en tutarlı ve güvenilir bir yaklaşım olarak, model geliştiriciler için iyi bir ilham kaynağı olabilir.

Local Rademacher Complexity Machine

Support Vector Machines (SVMs) are a state-of-the-art and powerful learning algorithm that can effectively solve many real world problems. SVMs are the transposition of the Vapnik–Chervonenkis (VC) theory into a learning algorithm. In this paper, we present the Local Rademacher Complexity Machine (LRCM), a transposition of the Local Rademacher Complexity (LRC) theory, the state-of-the-art evolution of the VC theory, into a learning algorithm. Analogously to what has been done for the SVMs, we will present first the theoretical ideas behind the LRC theory, we will show how these ideas can be translated into a learning algorithm, the LRCM, and then how the LRCM can be made efficient and kernelizable. By exploiting a series of real world datasets, we will show the effectiveness of the LRCM against the SVMs.

Permissible thyroid datasets assessment through kernel PC Algorithm and Vapnik Chervonenkis theory for categorization and classification

The proposed work assesses the composed mixture of machine learning algorithms(MLA) mentioned as principal component analysis (PCA) and kernel principal component analysis (KPCA) along with vapnik chervonenkis theory (VCT) on the datasets. The main hound is to trace the optimistic values obtained for classification and categorization of the datasets. In this induced methodology, PCA and KPCA are used for extraction of features, and VCT-KPCA is the proposed method which is used in categorization of features as an innovative hybrid approach (HA). The optimistic outcomes produced by the consistent datasets by PCA vary from the outcomes of KPCA, and subsequently, the hybrid proposal of VCT-KPCA method is indexed with high optimization ideals. The above work is considered on thyroid datasets which are from two different repositories UCI and e-IETD named as dataset-1(DS1) and dataset-2(DS2), respectively. To learn the differentiation between DS1 and DS2 canonical correlation analysis (CCA), weights in this act as an add-on procedure.

CC On/Off Scheduling Using Learning-Based Prediction for LTE in the Unlicensed Spectrum

In this letter, we propose a novel component carrier on/off scheduling scheme using the learning-based prediction for LTE in the unlicensed spectrum. Particularly, we use Lyapunov optimization to improve the performance of packet throughput and energy consumption, and use Vapnik-Chervonenkis theory to analyze the gap between the predicted optimal value and the practical one. The simulation results show that the proposed scheme can deliver good trade-off between packet throughput and energy consumption by turning off some carrier components, especially when the traffic load is not high.

Non-integer norm regularization SVM via Legendre–Fenchel duality

Support vector machine is an effective classification and regression method that uses VC theory of large margin to maximize the predictive accuracy while avoiding over-fitting of data. L2-norm regularization has been commonly used. If the training data set contains many noise features, L1-norm regularization SVM will provide a better performance. However, both L1-norm and L2-norm are not the optimal regularization method when handling a large number of redundant features and only a small amount of data points are useful for machine learning. We have therefore proposed an adaptive learning algorithm using the p-norm regularization SVM for 0<p≤2. Leveraging on the theory of Legendre–Fenchel duality, we derive a variational quadratic upper bound of non-differentiable non-convex Lp-norm regularized term when 0<p≤1. Generalization error bounds for non-integer norm regularization SVM were provided. Five cancer data sets from public data banks were used for the evaluation. All five evaluations empirically showed that the new adaptive algorithm was able to achieve the optimal prediction error using a less than L1 norm. On the seven different data sets having different sizes and different application domains, our approach was evaluated and compared to current state-of-the-art L1-norm and L2-norm SVM, repeatedly demonstrating that proposed method substantially improved performance. Moreover, we observed that the proposed p-norm penalty is more robust to noise features than the L1-norm and L2-norm penalties.

Indexability, concentration, and VC theory

Degrading performance of indexing schemes for exact similarity search in high dimensions has long since been linked to histograms of distributions of distances and other 1-Lipschitz functions getting concentrated. We discuss this observation in the framework of the phenomenon of concentration of measure on the structures of high dimension and the Vapnik–Chervonenkis theory of statistical learning.

Statistical learning theory: a tutorial

AbstractIn this article, we provide a tutorial overview of some aspects of statistical learning theory, which also goes by other names such as statistical pattern recognition, nonparametric classification and estimation, and supervised learning. We focus on the problem of two‐class pattern classification for various reasons. This problem is rich enough to capture many of the interesting aspects that are present in the cases of more than two classes and in the problem of estimation, and many of the results can be extended to these cases. Focusing on two‐class pattern classification simplifies our discussion, and yet it is directly applicable to a wide range of practical settings. We begin with a description of the two‐class pattern recognition problem. We then discuss various classical and state‐of‐the‐art approaches to this problem, with a focus on fundamental formulations, algorithms, and theoretical results. In particular, we describe nearest neighbor methods, kernel methods, multilayer perceptrons, Vapnik–Chervonenkis theory, support vector machines, and boosting. WIREs Comp Stat 2011 3 543–556 DOI: 10.1002/wics.179This article is categorized under: Statistical Learning and Exploratory Methods of the Data Sciences &gt; Clustering and Classification Statistical and Graphical Methods of Data Analysis &gt; Nonparametric Methods Statistical Learning and Exploratory Methods of the Data Sciences &gt; Pattern Recognition Statistical Learning and Exploratory Methods of the Data Sciences &gt; Knowledge Discovery Statistical Learning and Exploratory Methods of the Data Sciences &gt; Support Vector Machines Statistical Learning and Exploratory Methods of the Data Sciences &gt; Neural Networks

Another look at statistical learning theory and regularization

The paper reviews and highlights distinctions between function-approximation (FA) and VC theory and methodology, mainly within the setting of regression problems and a squared-error loss function, and illustrates empirically the differences between the two when data is sparse and/or input distribution is non-uniform. In FA theory, the goal is to estimate an unknown true dependency (or 'target' function) in regression problems, or posterior probability P(y/x) in classification problems. In VC theory, the goal is to 'imitate' unknown target function, in the sense of minimization of prediction risk or good 'generalization'. That is, the result of VC learning depends on (unknown) input distribution, while that of FA does not. This distinction is important because regularization theory originally introduced under clearly stated FA setting [Tikhonov, N. (1963). On solving ill-posed problem and method of regularization. Doklady Akademii Nauk USSR, 153, 501-504; Tikhonov, N., & V. Y. Arsenin (1977). Solution of ill-posed problems. Washington, DC: W. H. Winston], has been later used under risk-minimization or VC setting. More recently, several authors [Evgeniou, T., Pontil, M., & Poggio, T. (2000). Regularization networks and support vector machines. Advances in Computational Mathematics, 13, 1-50; Hastie, T., Tibshirani, R., & Friedman, J. (2001). The elements of statistical learning: Data mining, inference and prediction. Springer; Poggio, T. and Smale, S., (2003). The mathematics of learning: Dealing with data. Notices of the AMS, 50 (5), 537-544] applied constructive methodology based on regularization framework to learning dependencies from data (under VC-theoretical setting). However, such regularization-based learning is usually presented as a purely constructive methodology (with no clearly stated problem setting). This paper compares FA/regularization and VC/risk minimization methodologies in terms of underlying theoretical assumptions. The control of model complexity, using regularization and using the concept of margin in SVMs, is contrasted in the FA and VC formulations.

Energy Dissipation Bounds on Three Dimensional UWB Wireless Sensor Network

Considering a three dimensional TH-UWB wireless sensor network with n static identical randomly located sensor nodes in a sphere of volume normalized by one cubic meter, the formula for upper and lower bounds on the energy dissipation of some sensor node transmitting a data packet containing R bits to the sink node by multihop fashion are derived using Vapnik-Chervonenkis theory and Voroni tessellation covering sensor field. The main finding is that the upper and lower bounds on the energy dissipation are in inverse proportion to the node density n in the network. Thus, large-scale dense TH-UWB wireless sensor network is more preferable.

Nonparametric change detection and estimation in large-scale sensor networks

The problem of detecting changes in the distribution of alarmed sensors is considered. Under a nonparametric change detection framework, several detection and estimation algorithms are presented based on the Vapnik-Chervonenkis (VC) theory. Theoretical performance guarantees are obtained by providing error exponents for false-alarm and miss detection probabilities. Recursive algorithms for the efficient computation of test statistics are derived. The estimation problem is also considered in which, after detection is made, the location with maximum distribution change is estimated.

Model complexity control and statisticallearning theory

We discuss the problem of model complexity control also known as model selection. This problem frequently arises in the context of predictive learning and adaptive estimation of dependencies from finite data. First we review the problem of predictive learning as it relates to model complexity control. Then we discuss several issues important for practical implementation of complexity control, using the framework provided by Statistical Learning Theory (or Vapnik-Chervonenkis theory). Finally, we show practical applications of Vapnik-Chervonenkis (VC) generalization bounds for model complexity control. Empirical comparisons of different methods for complexity control suggest practical advantages of using VC-based model selection in settings where VC generalization bounds can be rigorously applied. We also argue that VC-theory provides methodological framework for complexity control even when its technical results can not be directly applied.

Pattern Recognition Theory in Nonlinear Signal Processing

A body of research has developed within the context of nonlinear signal and image processing that deals with the automatic, statistical design of digital window-based filters. Based on pairs of ideal and observed signals, a filter is designed in an effort to minimize the error between the ideal and filtered signals. The goodness of an optimal filter depends on the relation between the ideal and observed signals, but the goodness of a designed filter also depends on the amount of sample data from which it is designed. In order to lessen the design cost, a filter is often chosen from a given class of filters, thereby constraining the optimization and increasing the error of the optimal filter. To a great extent, the problem of filter design concerns striking the correct balance between the degree of constraint and the design cost. From a different perspective and in a different context, the problem of constraint versus sample size has been a major focus of study within the theory of pattern recognition. This paper discusses the design problem for nonlinear signal processing, shows how the issue naturally transitions into pattern recognition, and then provides a review of salient related pattern-recognition theory. In particular, it discusses classification rules, constrained classification, the Vapnik-Chervonenkis theory, and implications of that theory for morphological classifiers and neural networks. The paper closes by discussing some design approaches developed for nonlinear signal processing, and how the nature of these naturally lead to a decomposition of the error of a designed filter into a sum of the following components: the Bayes error of the unconstrained optimal filter, the cost of constraint, the cost of reducing complexity by compressing the original signal distribution, the design cost, and the contribution of prior knowledge to a decrease in the error. The main purpose of the paper is to present fundamental principles of pattern recognition theory within the framework of active research in nonlinear signal processing.

An introduction to kernel-based learning algorithms

This paper provides an introduction to support vector machines, kernel Fisher discriminant analysis, and kernel principal component analysis, as examples for successful kernel-based learning methods. We first give a short background about Vapnik-Chervonenkis theory and kernel feature spaces and then proceed to kernel based learning in supervised and unsupervised scenarios including practical and algorithmic considerations. We illustrate the usefulness of kernel algorithms by discussing applications such as optical character recognition and DNA analysis.

Learning pattern classification-a survey

Classical and recent results in statistical pattern recognition and learning theory are reviewed in a two-class pattern classification setting. This basic model best illustrates intuition and analysis techniques while still containing the essential features and serving as a prototype for many applications. Topics discussed include nearest neighbor, kernel, and histogram methods, Vapnik-Chervonenkis theory, and neural networks. The presentation and the large (though nonexhaustive) list of references is geared to provide a useful overview of this field for both specialists and nonspecialists.

Estimating Learning Curves of Concept Learning

In this paper, we describe an approximation method which enables us to study the average generalization performance of learning directly via hypothesis testing inequalities. This unites the learning and the hypothesis testing in a common viewpoint. In particular, we investigate learning curves of a so-called ill-disposed learning algorithm, which can provide useful implications regarding the problem of overfitting from a practical and yet scientific viewpoint. The learning problem is stated in the probably approximately correct (PAC) learning model, but only the average generalization performance is addressed. The resulting bounds are directly related to the number of system weights. The advantages of the theory are that it alleviates the practical pessimism frequently claimed for the results of the VC theory, and provides general insights. We illustrate the results with some numerical simulations. © 1997 Elsevier Science Ltd.

Towards more practical average bounds on supervised learning

In this paper, we describe a method which enables us to study the average generalization performance of learning directly via hypothesis testing inequalities. The resulting theory provides a unified viewpoint of average-case learning curves of concept learning and regression in realistic learning problems not necessarily within the Bayesian framework. The advantages of the theory are that it alleviates the practical pessimism frequently claimed for the results of the Vapnik-Chervonenkis (VC) theory and its alike, and provides general insights into generalization. Besides, the bounds on learning curves are directly related to the number of adjustable system weights. Although the theory is based on an approximation assumption, and cannot apply to the worst-case learning setting, the precondition of the assumption is mild, and the approximation itself is only a sufficient condition for the validity of the theory. We illustrate the results with numerical simulations, and apply the theory to examining the generalization ability of combination of neural networks.