When Ensemble Learning Meets Deep Learning: a New Deep Support Vector Machine for Classification

Abstract

Recently, Deep Learning (DL) method has received a significant breakthrough in the data representation, whose success mainly depends on its deep structure. In this paper, we focus on the DL research based on Support Vector Machine (SVM), and first present an Ex-Adaboost learning strategy, and then propose a new Deep Support Vector Machine (called DeepSVM). Unlike other DL algorithms based on SVM, in each layer, Ex-Adaboost is applied to not only select SVMs with the minimal error rate and the highest diversity, but also to produce the weight for each feature. In this way, new training data is obtained. By stacking these SVMs into multiple layers following the same way, we finally acquire a new set of deep features that can greatly boost the classification performance. In the end, the training data represented by these new features is regarded as the input for a standard SVM classifier. In the experimental part, we offer these answers to the following questions: 1) is the deep structure of DeepSVM really useful for classification problem? 2) Does Ex-Adaboost work, and is it helpful for further improving on DeepSVM’s performance with respect to the deep structure? 3) How much improvement in classification accuracy of DeepSVM, compared with other exist algorithms?