Reclassification of predictions for uncovering subgroup specific improvement

Abstract

Risk prediction models play an important role in prevention and treatment of several diseases. Models that are in clinical use are often refined and improved. In many instances, the most efficient way to improve a successful model is to identify subgroups for which there is a specific biological rationale for improvement and tailor the improved model to individuals in these subgroups, an approach especially in line with personalized medicine. At present, we lack statistical tools to evaluate improvements targeted to specific subgroups. Here, we propose simple tools to fill this gap. First, we extend a recently proposed measure, the Integrated Discrimination Improvement, using a linear model with covariates representing the subgroups. Next, we develop graphical and numerical tools that compare reclassification of two models, focusing only on those subjects for whom the two models reclassify differently. We apply these approaches to BRCAPRO, a genetic risk prediction model for breast and ovarian cancer, using data from MD Anderson Cancer Center. We also conduct a simulation study to investigate properties of the new reclassification measure and compare it with currently used measures. Our results show that the proposed tools can successfully uncover subgroup specific model improvements.