A Novel Bayesian Ensemble Pruning Method

Abstract

In ensemble learning, ensemble pruning is a procedure that aims at removing the unnecessary base classifiers and retaining the best subset of the base classifiers. We presented a two-step ensemble pruning framework, in which the optimal size of the pruned ensemble is first decided, and then with the optimal size as input, the optimal ensemble is selected. For the first step to find the optimal ensemble size, we presented an algorithm that can be proved to be able to find the Bayesian optimal ensemble size. For the second step, we developed two greedy forward pruning methods, i.e., the Bayesian Pruning (BP) method and the Bayesian Independent Pruning (BIP) method. In the BP method, we assumed that the probability of a candidate ensemble to be the optimal ensemble follows the Generalized Beta distribution. And in the BIP method, we further assumed that whether a base classifier belongs to the optimal ensemble is independent to the other base classifiers. Experimental results on twenty benchmark data sets showed that the BP and BIP methods achieved competitive performance in contrast to other state-of-the-art algorithms.