Quantifying uncertainty of subsampling-based ensemble methods under a U-statistic framework

Abstract

ABSTRACT This paper addresses the problem of variance estimation of predictions obtained from a subsampling-based ensemble estimator, such as subbagging and sub-random forest. We first recognize that a subsampling-based ensemble can be written as an infinite-order U-statistic of degree , where is the subsample size that may depend on the learning sample size n. As a result, one can study the uncertainty of predictions obtained from a subsampling-based ensemble under a U-statistic framework, such as approximating its asymptotic variance. However, existing methods used to estimate the asymptotic variance relies on some regularity conditions. In addition, they tend to yield variance estimations with large bias in finite sample scenarios. Motivated by the work of Wang and Lindsay (2014), we propose to construct an unbiased variance estimator for a subsampling-based ensemble. It is efficient to realize with the help of a partition-resampling scheme. We show by simulation studies that the proposed variance estimator yields better performance in terms of mean, standard deviation, and mean squared error compared to both the infinitesimal jackknife and internal variance estimation methods under either a simple linear regression model or a multivariate adaptive regression splines model. Furthermore, we present how to construct an asymptotic confidence interval for the expected prediction at a given test instance using the proposed variance estimator, and compare its coverage probability to that of competing methods. In the end, we demonstrate the practical applications of the methodology using two real data examples.