Classifier ensemble methods in feature selection

Abstract

Feature selection has become an indispensable preprocessing step in an expert system. Improving the feature selection performance could guide such a system to make better decisions. Classifier ensembles are known to improve performance when compared to the use of a single classifier. In this study, we aim to perform a formal comparison of different classifier ensemble methods on the feature selection domain. For this purpose, we compare the performances of six classifier ensemble methods: a greedy approach, two average-based approaches, two majority voting approaches, and a meta-classifier approach. In our study, the classifier ensemble involves five machine learning techniques: Logistic Regression, Support Vector Machines, Extreme Learning Machine, Naïve Bayes, and Decision Tree. Experiments are carried on 12 well-known datasets, and results with statistical tests are provided. The results indicate that ensemble methods perform better than single classifiers, yet, they require a longer execution time. Moreover, they can minimize the number of features better than existing ensemble algorithms, namely Random Forest, AdaBoost, and Gradient Boosting, in a less amount of time. Among ensemble methods, the greedy based method performs well in terms of both classification accuracy and execution time.