A method to measure predictive ability of an injury risk curve using an observation-adjusted area under the receiver operating characteristic curve

Abstract

Area under the receiver operating characteristic curve (AROC) is commonly used to choose a biomechanical metric from which to construct an injury risk curve (IRC). However, AROC may not handle censored datasets adequately. Survival analysis creates robust estimates of IRCs which accommodate censored data. We present an observation-adjusted ROC (oaROC) which uses the survival-based IRC to estimate the AROC. We verified and evaluated this method using simulated datasets of different censoring statuses and sample sizes. For a dataset with 1000 left and right censored observations, the median AROC closely approached the oaROCTrue, or the oaROC calculated using an assumed “true” IRC, differing by a fraction of a percent, 0.1%. Using simulated datasets with various censoring, we found that oaROC converged onto oaROCTrue in all cases. For datasets with right and non-censored observations, AROC did not converge onto oaROCTrue. oaROC for datasets with only non-censored observations converged the fastest, and for a dataset with 10 observations, the median oaROC differed from oaROCTrue by 2.74% while the corresponding median AROC with left and right censored data differed from oaROCTrue by 9.74%. We also calculated the AROC and oaROC for a published side impact dataset, and differences between the two methods ranged between −24.08% and 24.55% depending on metric. Overall, when compared with AROC, we found oaROC performs equivalently for doubly censored data, better for non-censored data, and can accommodate more types of data than AROC. While more validation is needed, the results indicate that oaROC is a viable alternative which can be incorporated into the metric selection process for IRCs.