Minimal Complexity Machine Research Articles

Kernel optimization using conformal maps for the minimal complexity machine

The Minimal Complexity Machine (MCM) is a kernel-based learning model that can learn very sparse models that yield comparable or better performance than Support Vector Machines (SVMs). However, kernel optimization for the MCM has not yet been explored. It has been shown in prior work that a data dependent kernel helps improve generalization. We show results on data dependent optimized kernels for the MCM and a large-scale MCM variant. Results on benchmark datasets demonstrate both model sparsity and improved generalization.

Neurodynamical classifiers with low model complexity

The recently proposed Minimal Complexity Machine (MCM) finds a hyperplane classifier by minimizing an upper bound on the Vapnik–Chervonenkis (VC) dimension. The VC dimension measures the capacity or model complexity of a learning machine. Vapnik’s risk formula indicates that models with smaller VC dimension are expected to show improved generalization. On many benchmark datasets, the MCM generalizes better than SVMs and uses far fewer support vectors than the number used by SVMs. In this paper, we describe a neural network that converges to the MCM solution. We employ the MCM neurodynamical system as the final layer of a neural network architecture. Our approach also optimizes the weights of all layers in order to minimize the objective, which is a combination of a bound on the VC dimension and the classification error. We illustrate the use of this model for robust binary and multi-class classification. Numerical experiments on benchmark datasets from the UCI repository show that the proposed approach is scalable and accurate, and learns models with improved accuracies and fewer support vectors.

QMCM: Minimizing Vapnik’s bound on the VC dimension

The recently proposed Minimal Complexity Machine (MCM) learns a hyperplane classifier by minimizing a bound on the Vapnik-Chervonenkis (VC) dimension. Both the linear and kernel versions of the MCM solve a linear programming problem, in order to minimize a bound on the VC dimension. This paper proposes a new quadratic programming formulation, termed as the Quadratic MCM (QMCM), that minimizes a tighter bound on the VC dimension. We present two variants of the QMCM, that differ in the norm of the error vector being minimized. We also explore a scalable variant of the QMCM for large datasets using Stochastic Gradient Descent (SGD), and present the use of the QMCM as a viable featureselection method, in view of the the sparse nature of the models it learns. We compare the performance of the QMCM variants with LIBLinear, a linear Support Vector Machine (SVM) library; as well as against Pegasos and the linear MCM for large datasets, along with sequential feature selection methods and ReliefF. Our results validate the superiority of the QMCM in terms of statistically significant improvements on benchmark datasets from the UCI Machine Learning repository.

Generalized Pinball Loss SVMs

Abstract Support Vector Machine (SVM) is a well-known efficient classification technique which appropriately trade-offs between the training error and the generalization ability of the classifier. But SVM’s are known to be sensitive towards noise and not stable with respect to re-sampling. Recently, Huang et al. (2014) proposed a novel SVM formulation with pinball loss function (Pin-SVM) model to handle noise sensitivity and instability to re-sampling. The Pin-SVM model in its pure form loses sparsity and therefore Huang et al. (2014) proposed a new pinball SVM model termed as ϵ-insensitive zone Pin-SVM. The ϵ-insensitive zone Pin-SVM formulation requires the value of ϵ to be pre-specified. Taking motivation from these developments, we propose a modified (ϵ1, ϵ2)-insensitive zone Pin-SVM ((ϵ1, ϵ2)-Mod-Pin-SVM) model in which the asymmetric spread of insensitive zone is optimized and therefore it is data-driven. An interesting feature of our proposed (ϵ1, ϵ2)-Mod-Pin-SVM model is that it appropriately trade-offs the generalization ability, the sparsity as well as the noise insensitivity. The (ϵ1, ϵ2)-Mod-Pin-SVM model is very general and subsumes a number of popular SVM type models. Its relation with Minimal Complexity Machine (MCM) introduced by Jayadeva (2015) is interesting and novel. The experimental results on several benchmark datasets prove the effectiveness of our proposed (ϵ1, ϵ2)-Mod-Pin-SVM formulation to that of other state-of-the-art classification algorithms.

Ultra-Sparse Classifiers Through Minimizing the VC Dimension in the Empirical Feature Space

Sparse representations have gained much interest due to the rapid growth of intelligent embedded systems, the need to reduce the time to mine large datasets, and for reducing the footprint of recognition based applications on portable devices. Computational learning theory tells us that the Vapnik–Chervonenkis (VC) dimension of a learning model directly impacts the structural risk and the generalization ability of a learning model. The minimal complexity machine (MCM) was recently proposed as a way to learn a hyperplane classifier by minimizing a tight bound on the VC dimension; results show that it learns very sparse representations that yield test set accuracies that are comparable to the state-of-the-art. The MCM formulation works in the primal itself, both when the classifier is learnt in the input space and when it is learnt implicitly in a higher dimensional feature space. In the latter case, the hyperplane is constructed in the empirical feature space (EFS). In this paper, we examine the hyperplane restricted to the EFS. The EFS is a finite dimensional vector space spanned by the image vectors in a higher dimensional feature space. Since the VC dimension of a linear hyperplane classifier is exactly the number of features used by the classifier, the dimension of the EFS is a direct measure of both the sparsity of the model and the VC dimension. This allows us to formulate optimization problems that focus on learning sparse representations, and yet generalize well. We derive an EFS version of the MCM, that allows us to minimize the model complexity and improve sparsity. We also propose a novel least squares version of the MCM in the EFS. Experimental results demonstrate that the EFS variants yield sparse models with generalization comparable to the state-of-the-art.

Sparse short-term time series forecasting models via minimum model complexity

Time series forecasting is of fundamental importance in big data analysis. The prediction of noisy, non-stationary, and chaotic time series demands good generalization from small amounts of data. Vapnik showed that the total risk is dependent on both, empirical error as well as model complexity, where the latter may be measured in terms of the Vapnik–Chervonenkis (VC) dimension. In other words, good generalization requires minimizing model complexity. The recently proposed Minimal Complexity Machine (MCM) has been shown to minimize a tight bound on the VC dimension, and has further been extended to Minimal Complexity Machine Regression (MCMR). In this paper, we present an original approach based on the MCM regressor, which builds sparse and accurate models for short-term time series forecasting. Results on a number of datasets establish that the proposed approach is superior to a number of state-of-the-art methods, and yields sparse models. These sparse models are able to extract only the most important information present in the data sets, thereby achieving high accuracy. Sparsity in time series forecasting models is also important in reducing the evaluation time. This assumes importance when the models need to be evaluated in real time, such as when they are used as part of trading flows.

Characterizing objects with SIKA features for multiclass classification

This paper presents a novel approach for multiclass classification by fusion of KAZE and Scale Invariant Feature Transform (SIFT) features followed by Minimal Complexity Machine (MCM) as the classifier. Unlike the existing features, the paper proposes a new feature SIKA to represent characteristics of an object, as opposed to just forming a compendium of interest points in an image to represent the object characteristics. Further we append a strong and lightweight classifier, MCM to the technique. The resulting classifier easily outperforms existing techniques based on handcrafted features. Two new scores Keypoint Overlap Score (KOS) and Mean Keypoint Overlap Score (MKOS) have also been proposed as part of this work which are useful in establishing the strength of features for object representation.

Learning a hyperplane regressor through a tight bound on the VC dimension

In this paper, we show how to learn a hyperplane regressor by minimizing a tight or Θ bound on its VC dimension. While minimizing the VC dimension with respect to the defining variables is an ill posed and intractable problem, we propose a smooth, continuous, and differentiable function for a tight bound. Minimizing a tight bound yields the Minimal Complexity Machine (MCM) Regressor, and involves solving a simple linear programming problem. Experimental results show that on a number of benchmark datasets, the proposed approach yields regressors with error rates much lower than those obtained with conventional SVM regresssors, while often using fewer support vectors. On some benchmark datasets, the number of support vectors is less than one-tenth the number used by SVMs, indicating that the MCM does indeed learn simpler representations.

Learning a hyperplane classifier by minimizing an exact bound on the VC dimension1

The VC dimension measures the capacity of a learning machine, and a low VC dimension leads to good generalization. While SVMs produce state-of-the-art learning performance, it is well known that the VC dimension of a SVM can be unbounded; despite good results in practice, there is no guarantee of good generalization. In this paper, we show how to learn a hyperplane classifier by minimizing an exact, or \boldmath{$\Theta$} bound on its VC dimension. The proposed approach, termed as the Minimal Complexity Machine (MCM), involves solving a simple linear programming problem. Experimental results show, that on a number of benchmark datasets, the proposed approach learns classifiers with error rates much less than conventional SVMs, while often using fewer support vectors. On many benchmark datasets, the number of support vectors is less than one-tenth the number used by SVMs, indicating that the MCM does indeed learn simpler representations.