Using the area under an estimated ROC curve to test the adequacy of binary predictors

Abstract

ABSTRACTWe consider using the area under an empirical receiver operating characteristic curve to test the hypothesis that a predictive index combined with a range of cutoffs performs no better than pure chance in forecasting a binary outcome. This corresponds to the null hypothesis that the area in question, denoted as AUC, is 1/2. We show that if the predictive index comes from a first-stage regression model estimated over the same data set, then testing the null based on the standard asymptotic normality results leads to severe size distortion in general settings. We then analytically derive the proper asymptotic null distribution of the empirical AUC in a special case; namely, when the first-stage regressors are Bernoulli random variables. This distribution can be utilised to construct a fully in-sample test of with correct size and more power than out-of-sample tests based on sample splitting, though practical application becomes cumbersome with more than two regressors.