Optimal Margin Distribution Machine with Sparsity Inducing Penalty

Abstract

Recently a promising research direction of statistical learning has been advocated, i.e., the optimal margin distribution learning, with the central idea of optimizing the margin distribution. As the most representative approach of this new learning paradigm, the optimal margin distribution machine (ODM) considers maximizing the margin mean and minimizing the margin variance simultaneously. The standard ODM exploits the l_2-norm penalty, which gives rise to a dense decision boundary. However, in some situations, the model with parsimonious representation is more preferred, due to the redundant noisy features or limited computing resources. In this paper, we propose the sparse optimal margin distribution machine (Sparse ODM), which aims to achieve better generalization performance with moderate model size. For optimization, we extends an efficient coordinate descent method to solve the final problem since the variables are decoupled. In each iteration, we propose a modified Newton method to solve the one-variable sub-problem. Experimental results on both synthetic and real data sets show the superiority of the proposed method.