Filter-based optimization techniques for selection of feature subsets in ensemble systems

Abstract

Feature selection methods select a subset of attributes (features) of a dataset and it is done based on a defined measure, eliminating the redundant and irrelevant ones. When a feature selection method is applied in a dataset, we aim to improve the quality of the dataset representation. For ensemble systems, feature selection techniques can supply different feature subsets for the individual components, reducing the redundancy that can exist among the features of an input pattern and to increase the diversity level of these systems. This paper proposes the application of three well-known optimization techniques (particle swarm optimization, ant-colony optimization and genetic algorithms), in both mono and bi-objective versions, to choose subsets of features for the individual components of ensembles. The feature selection process was based on two filter-based evaluation criteria that tried to capture the idea of diversity of individual classifiers and group diversity of an ensemble system. In this case, these optimization techniques try to maximize these diversities measures, either individually (mono-objective) or together (bi-objective). An empirical analysis was performed, where all ensemble systems were applied to 11 datasets and we compared both mono and bi-objective versions among each other and with a random subset procedure. Based on the empirical analysis, we will observe that PSO with a bi-objective function will be the most promising direction, when selecting attributes for individual components of ensemble systems.