Discrimination Index, the Area Under the ROC Curve

Abstract

The accuracy of fit of a mathematical predictive model is the degree to which the predicted values coincide with the observed outcome. When the outcome variable is dichotomous and predictions are stated as probabilities that an event will occur, models can be checked for good discrimination and calibration. In case of the multiple logistic regression model for binary outcomes (event, non-event), the area under the ROC (Receiver Operating Characteristic) curve is the most used measure of model discrimination. The area under the ROC curve is identical to the Mann-Whitney statistic. We consider shift models for the distributions of predicted probabilities for event and non-event. From the interval estimates of the shift parameter, we calculate the confidence intervals for the area under the ROC curve. Also, we present the development of a general description of an overall discrimination index C (overall C) which we can extend to a survival time model such as the the Cox regression model. The general theory of rank correlation is applied in developing the overall C. The overall C is a linear combination of three independent components: event vs. non-event, event vs. event and event vs. censored. By showing that these three components are asymptotically normally distributed, the overall C is shown to be asymptotically normally distributed. The expected value and the variance of the overall C are presented.