Selecting the right-size model for prediction

Abstract

We evaluate the effectiveness of cross-validation in selecting the right-size model for decision tree and k-nearest neighbor learning methods. For samples with at least 200 cases, extensive empirical evidence supports the following conclusions relative to complexity-fit selection: (a) 10-fold cross-validation is nearly unbiased; (b) ignoring model complexity-fit and picking the “standard” model is highly biased; (c) 10-fold cross-validation is consistent with optimal complexity-fit selection for large sample sizes and (d) the accuracy of complexity-fit selection by 10-fold cross-validation is largely dependent on sample size, irrespective of the population distribution.