Block-regularized repeated learning-testing for estimating generalization error

Abstract

Repeated learning-testing (RLT) is a popular cross-validation method that is usually adopted in estimation and comparison of algorithm performance in machine learning. However, the variance of the estimator of the generalization error in the RLT method is easily affected by random partitions. Poor data partitioning may cause a large overlap between any two training sets of RLT and enlarge the variance in the RLT estimator of the generalization error. Thus, in this study, we constrain numbers of overlapping samples between any two training sets and construct a novel data partitioning schema in which RLT is called the block-regularized RLT (BRLT). We theoretically prove that the BRLT estimator has a smaller variance than does the RLT estimator. Specifically, the variance of the RLT estimator reaches its minimum when samples in a data set equally occur in all training sets and all the numbers of overlapping samples are equal. Furthermore, we provide easy-to-use construction algorithms of BRLT’s partition set for several training set size and partition count settings by adopting two-level orthogonal arrays. We also illustrate BRLT’s optimal properties with several simulated and real-world examples.