A novel ensemble method for k-nearest neighbor

Abstract

In this paper, to address the issue that ensembling k-nearest neighbor (kNN) classifiers with resampling approaches cannot generate component classifiers with a large diversity, we consider ensembling kNN through a multimodal perturbation-based method. Since kNN is sensitive to the input attributes, we propose a weighted heterogeneous distance Metric (WHDM). By using a WHDM and evidence theory, a progressive kNN classifier is developed. Based on a progressive kNN, the random subspace method, attribute reduction, and Bagging, a novel algorithm termed RRSB (reduced random subspace-based Bagging) is proposed for construct ensemble classifier, which can increase the diversity of component classifiers without damaging the accuracy of the component classifiers. In detail, RRSB adopts the perturbation on the learning parameter with a weighted heterogeneous distance metric, the perturbation on the input space with random subspace and attribute reduction, the perturbation on the training data with Bagging, and the perturbation on the output target of k neighbors with evidence theory. In the experimental stage, the value of k, the different perturbations on RRSB and the ensemble size are analyzed. In addition, RRSB is compared with other multimodal perturbation-based ensemble algorithms on multiple UCI data sets and a KDD data set. The results from the experiments demonstrate the effectiveness of RRSB for kNN ensembling.