Abstract
Heterogeneous ensembles consist of predictors of different types, which are likely to have different biases. If these biases are complementary, the combination of their decisions is beneficial and could be superior to homogeneous ensembles. In this paper, a family of heterogeneous ensembles is built by pooling classifiers from M homogeneous ensembles of different types of size T. Depending on the fraction of base classifiers of each type, a particular heterogeneous combination in this family is represented by a point in a regular simplex in M dimensions. The M vertices of this simplex represent the different homogeneous ensembles. A displacement away from one of these vertices effects a smooth transformation of the corresponding homogeneous ensemble into a heterogeneous one. The optimal composition of such heterogeneous ensemble can be determined using cross-validation or, if bootstrap samples are used to build the individual classifiers, out-of-bag data. The proposed heterogeneous ensemble building strategy, composed of neural networks, SVMs, and random trees (i.e. from a standard random forest), is analyzed in a comprehensive empirical analysis and compared to a benchmark of other heterogeneous and homogeneous ensembles. The achieved results illustrate the gains that can be achieved by the proposed ensemble creation method with respect to both homogeneous ensembles and to the tested heterogeneous building strategy at a fraction of the training cost.
Highlights
Building an effective classifier for a specific problem is a difficult task
In this work we propose to analyze heterogeneous ensembles in which the individual classifiers are selected from homogeneous ensembles
This problem is related to the Matrix Cover problem (MC) [25], in which the rows and columns correspond to the decisions made by each base learners hi ∈ T for each data point n of a training set D = {xn, yn}Nl=tr1ain, respectively
Summary
Building an effective classifier for a specific problem is a difficult task. To be successful, a variety of aspects need to be taken into account: the data structure, the information that can be used for prediction, the number of the labeled examples available for induction, the noise level, among others. Since different learning algorithms are used to generate the base learners, heterogeneous ensembles are intrinsically diverse In this case, the main difficulty resides in determining the optimal way to combine the predictions of the different models in the ensemble. In a recent study [23], to build an effective heterogeneous combination, the authors trim the base learners with poor performance so that only optimal classifiers will be preserved in the ensemble. In another study [24], which to the best of our knowledge is one of the most related works to ours, the authors used a differential evolution algorithm to optimize the weighting votes of diverse base learns in a heterogeneous ensemble They used the average Matthews Correlation Coefficient (MCC), calculated over 10-fold crossvalidation, to evaluate each combination and obtain the base learners’ optimal voting weights.
Sign-in/Register to view highlights & summary 
Published Version (
Free)
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call