The Effectiveness of a New Negative Correlation Learning Algorithm for Classification Ensembles

Abstract

In an earlier paper, we proposed a new negative correlation learning (NCL) algorithm for classification ensembles, called AdaBoost.NC, which has significantly better performance than the standard AdaBoost and other NCL algorithms on many benchmark data sets with low computation cost. In this paper, we give deeper insight into this algorithm from both theoretical and experimental aspects to understand its effectiveness. We explain why AdaBoost.NC can reduce error correlation within the ensemble and improve the classification performance. We also show the role of the $amb$ (penalty) term in the training error. Finally, we examine the effectiveness of AdaBoost.NC by varying two pre-defined parameters – penalty strength $\lambda$ and ensemble size T. Experiments are carried out on both artificial and real-world data sets, which show that AdaBoost.NC does produce smaller error correlation along with training epochs, and a lower test error comparing to the standard AdaBoost. The optimal $\lambda$ depends on problem domains and base learners. The performance of AdaBoost.NC becomes stable as T gets larger. It is more effective when T is comparatively small.