Novel method of estimating AROC using an injury risk curve for biomechanical injury metric selection

Abstract

ABSTRACTObjective: Area under the receiver operating characteristic (AROC) is commonly used to evaluate an injury metric's ability to discriminate between injury and noninjury cases. However, AROC has limitations and may not handle censored data sets adequately. Survival methodology creates robust estimates of injury risk curves (IRCs) which accommodate censored data. We developed an observation-adjusted ROC (oaROC), an AROC-like statistic calculated from the IRC.Methods: oaROC uses an observational distribution and an IRC to measure true positive rate (TPR) and false positive rate (FPR). The oaROC represents what the AROC would be with a large number of observations sampled from the IRC. We verified this using a limit test with simulated data sets at various sample sizes drawn from an assumed “true” IRC. For each sample size, 5,000 different data sets were created; a conventional AROC was calculated for each data set and compared with the single oaROC, which was calculated from the “true” IRC and not dependent on sample size.Results: The oaROC, calculated from the simulated IRC, was 0.911. At a sample size of 20, the mean AROC was 0.930 (2.0% difference). At a sample size of 1,000, the mean AROC was 0.9114 (0.02% difference).Conclusion: We verified that AROC approaches the oaROC with increasing sample sizes, and oaROC presents a measure of IRC discriminatory ability. Survival methodology can estimate IRCs using censored observations and the oaROC was designed with this in mind. The oaROC may be a useful measure of discrimination for data sets containing censored data. Further investigation is needed to evaluate oaROC calculated from estimated IRCs.