PMA12: RELATIONSHIP BETWEEN NONPARAMETRIC RECEIVER OPERATING CHARACTERISTIC ANALYSIS AND A LIKELIHOOD-RATIO TEST FOR MODEL SELECTION: I. A MONTE CARLO SIMULATION USING CONTINUOUS DATA

Abstract

BACKGROUND: In a simulation using continuous data, we compared the performance of DeLong's test for nonparametric receiver operating characteristic (ROC) curves (Biometrics 1988; 44: 837–845) with that of Vuong's test for model selection (Econometrica 1989; 57: 307–333). Both tests were found to yield similar results regardless of sample size. Diagnostic tests are often measured on an ordinal rating scale, and nonparametric methods tend to underestimate the area under the ROC curve (AUC) when used with discrete data. Thus, it was conceivable that level of measurement might influence the performance of DeLong's test. OBJECTIVE: A second Monte Carlo simulation was performed to determine whether DeLong's and Vuong's tests behave differently when used with discrete data. METHODS: One thousand observations were randomly generated for a Bernoulli dependent variable and 11 binomial independent variables. The independents were generated such that realizations were integers ranging from 1–10. Bootstrapped estimates for AUC and logistic regression model log-likelihood (LL) were derived using 1000 replications of sample size 10, 25, 50, 100, 175, 250, and 500. At each sample size, predictors were compared on the basis of AUC using DeLong's test and model LL using Vuong's test. The random number seed was set so that identical samples were compared with each test. RESULTS: In general, the two tests yielded similar statistical conclusions. Asymptotically, the observed power of Vuong's test was greater than that of DeLong's test. In smaller samples, however, Vuong's test was slightly less powerful. The results of the two tests diverged in only three cases in small samples. CONCLUSIONS: The results of this analysis correspond to those of our simulation using continuous data. Though slightly less powerful than DeLong's test, Vuong's test is more flexible and is less time consuming. Given the results of both simulations, Vuong's test appears to present a useful alternative to ROC analysis for comparing the accuracy of diagnostic tests.