Non-integer norm regularization SVM via Legendre–Fenchel duality

Abstract

Support vector machine is an effective classification and regression method that uses VC theory of large margin to maximize the predictive accuracy while avoiding over-fitting of data. L2-norm regularization has been commonly used. If the training data set contains many noise features, L1-norm regularization SVM will provide a better performance. However, both L1-norm and L2-norm are not the optimal regularization method when handling a large number of redundant features and only a small amount of data points are useful for machine learning. We have therefore proposed an adaptive learning algorithm using the p-norm regularization SVM for 0<p≤2. Leveraging on the theory of Legendre–Fenchel duality, we derive a variational quadratic upper bound of non-differentiable non-convex Lp-norm regularized term when 0<p≤1. Generalization error bounds for non-integer norm regularization SVM were provided. Five cancer data sets from public data banks were used for the evaluation. All five evaluations empirically showed that the new adaptive algorithm was able to achieve the optimal prediction error using a less than L1 norm. On the seven different data sets having different sizes and different application domains, our approach was evaluated and compared to current state-of-the-art L1-norm and L2-norm SVM, repeatedly demonstrating that proposed method substantially improved performance. Moreover, we observed that the proposed p-norm penalty is more robust to noise features than the L1-norm and L2-norm penalties.