Leveraging Κ-nn for generic classification boosting

Abstract

k-nearest neighbors (k-NN) voting rules are an effective tool in countless many machine learning techniques. In spite of its simplicity, k-NN classification is very attractive to practitioners, as it has shown very good performances in practical applications. However, it suffers from various drawbacks, like sensitivity to “noisy” prototypes and poor generalization properties when dealing with sparse, high-dimensional data. In this paper, we tackle the k-NN classification problem at its core by providing a novel k-NN boosting approach. We propose a Universal Nearest Neighbors (UNN) algorithm, which induces a leveraged k-NN rule by globally minimizing a surrogate risk upper bounding the empirical misclassification rate over training data. Interestingly, this surrogate risk can be arbitrary chosen from a class of Bregman loss functions, including the familiar exponential, logistic and squared losses. Furthermore, we show that UNN allows to efficiently filter a dataset of instances by keeping only a small fraction of data. Experimental results on the synthetic Ripley's dataset show that such a filtering strategy is able to reject “noisy” examples, and yields a classification error close to the optimal Bayes error. Experiments on standard UCI datasets show significant improvements over the current state of the art.