Newton-based approach to solving K-SVCR and Twin-KSVC multi-class classification in the primal space

Abstract

Multi-class classification is an important problem in machine learning, which often occurs in the real world and is an ongoing research issue. Support vector classification-regression machine for k-class classification (K-SVCR) and twin k-class support vector classification (Twin-KSVC) are two novel machine learning methods for multi-class classification problems. This paper presents novel methods to solve the primal problems of K-SVCR and Twin-KSVC, known as NK-SVCR and NTW-KSVC, respectively. The proposed methods evaluate all training data into a “1-versus-1-versus-rest” structure, so it generates ternary outputs {−1,0,+1}. The primal problems are reformulated as unconstrained optimization problems so that the objective functions are only once differentiable, not twice, therefore an extension of the Newton-Armijo algorithm is adopted for finding their solution. To test the efficiency and validity of the proposed methods, we compare the classification accuracy and learning time of these methods with K-SVCR and Twin-KSVC on the United States Postal Service (USPS) handwriting digital data sets and several University of California Irvine (UCI) benchmark data sets. To analyze more in aspect of training time, we also compared all methods on the multi-class version of the Normally Distributed Clustered (NDC) database. To further analyze classification accuracy and learning time differences between the classifiers, the statistical Friedman’s test is used.