Convolution smoothing and non-convex regularization for support vector machine in high dimensions

Abstract

The support vector machine (SVM) is a well-known statistical learning tool for binary classification. One serious drawback of SVM is that it can be adversely affected by redundant variables, and research has shown that variable selection is crucial and necessary for achieving good classification accuracy. Hence some SVM variable selection studies have been devoted, and they have an unified “empirical hinge loss plus sparse penalty” formulation. However, a noteworthy issue is the computational complexity of existing methods is high especially for large-scale problems, due to the non-smoothness of the hinge loss. To solve this issue, we first propose a convolution smoothing approach, which turns the non-smooth hinge loss into a smooth surrogate one, and they are asymptotically equivalent. Moreover, we construct computationally more efficient SVM variable selection procedure by implementing non-convex penalized convolution smooth hinge loss. In theory, we prove that the resulting variable selection possesses the oracle property when the number of predictors is diverging. Numerical experiments also confirm the good performance of the new method.