Model selection for k-nearest neighbors regression using VC bounds

Abstract

We discuss an analytic model selection for k-nearest neighbors regression method using VC generalization bounds. Whereas existing implementations of k-nn regression estimate the model complexity as n/k, where n is the number of samples, we propose a new model complexity estimate. The proposed new complexity index used as the VC-dimension in VC bounds yields a new analytic method for model selection. Empirical results for low dimensional and high dimensional data sets indicate that the proposed model selection approach provides accurate model selection that is consistently better than the previously used complexity measure. In fact, prediction accuracy of the proposed analytic method is similar to resampling (cross-validation) approach for optimal selection of k.