Abstract
ObjectiveImprovements in clinical risk prediction models for osteoporosis-related fracture can be evaluated using area under the receiver operating characteristic (AUROC) curve and calibration, as well as reclassification statistics such as the net reclassification improvement (NRI) and integrated discrimination improvement (IDI) statistics. Our objective was to compare the performance of these measures for assessing improvements to an existing fracture risk prediction model. We simulated the effect of a new, randomly-generated risk factor on prediction of major osteoporotic fracture (MOF) for the internationally-validated FRAX® model in a cohort from the Manitoba Bone Mineral Density (BMD) Registry.ResultsThe study cohort was comprised of 31,999 women 50+ years of age; 9.9% sustained at least one MOF in a mean follow-up of 8.4 years. The original prediction model had good discriminative performance, with AUROC = 0.706 and calibration (ratio of observed to predicted risk) of 0.990. The addition of the simulated risk factor resulted in improvements in NRI and IDI for most investigated conditions, while AUROC decreased and changes in calibration were negative. Reclassification measures may give different information than discrimination and calibration about the performance of new clinical risk factors.
Highlights
Methods to predict the risk of an outcome are receiving considerable attention in the clinical literature
This is an important topic for osteoporosis-related fracture risk prediction, where a number of models have been proposed [2] and numerous studies have examined the incremental improvement in prediction when biomarkers or other clinical characteristics of patients are introduced to existing models [3, 4]
Clinicians have given increased attention to reclassification tables and statistics such as the net reclassification index (NRI), which summarize the change in risk probability or the frequency of individuals who will move from one risk category to another based on the addition of a new risk factor to the original prediction model
Summary
Study cohort characteristics The study cohort (Table 1) was comprised of 31,999 women 50+ years of age. Risk prediction model characteristics and computer simulation The estimated AUROC of the original model was 0.706 [95% confidence interval (95% CI) 0.697–0.716] and calibration was 0.990. Using this model, 6.8% of cohort members were predicted to have low fracture risk (i.e., < 10%), 13.9% as moderate fracture risk (i.e., 10–20%), and 27.3% as high fracture risk (i.e., > 20%). The second set of results was obtained when the prevalence of the new simulated risk factor was varied and other simulation parameters were held constant. Given that the IDI is based on continuous values of the risk probabilities, it did not change with variations in the intervention threshold, nor did the AUROC and calibration statistics
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
