Statistical flaws in the development of a prediction model

Abstract

While I read with great interest the development of a nomogram to predict the 5-year recurrence of border ovarian tumors, my enthusiasm was tempered by the numerous deficiencies in the statistical analysis.1Bendifallah S. Ballester M. Uzan C. Fauvet R. Morice P. Darai E. Nomogram to predict recurrence in patients with early- and advanced-stage mucinous and serous borderline ovarian tumors.Am J Obstet Gynecol. 2014; 211: 637.e1-637.e6Scopus (27) Google Scholar The first issue is the prediction time horizon, which was reported to be 5 years. All included patients appear to have been followed for a minimum 5 years from diagnosis to recurrence or last follow up, with a median of 7.9 years; it is therefore improbable that recurrence at 5 years is actually being predicted. To predict recurrence at 5 years would require follow-up evaluation until 5 years from diagnosis, with those lost to follow up censored at the point of last known outcome status. Survival-based methods such as Cox regression are then used, which uses all available information, in preference to logistic regression, which was used by Bendifallah et al,1Bendifallah S. Ballester M. Uzan C. Fauvet R. Morice P. Darai E. Nomogram to predict recurrence in patients with early- and advanced-stage mucinous and serous borderline ovarian tumors.Am J Obstet Gynecol. 2014; 211: 637.e1-637.e6Scopus (27) Google Scholar because logistic regression is unable to account for censored observations. This naturally brings us to the absence of information on whether there were patients who were followed for <5 years or had a recurrence before the 5 years and were omitted from the analysis. This would seem not only illogical, given the aim to predict recurrence by 5 years, but also likely to suggest selection bias. Bootstrapping (or cross-validation) is seen as the preferred method for internal validation to quantify and adjust for optimism. However, all variable selection procedures, including the flawed univariate screening2Sun G.W. Shook T.L. Kay G.L. Inappropriate use of bivariable analysis to screen risk factors for use in multivariable analysis.J Clin Epidemiol. 1996; 49: 907-916Abstract Full Text PDF PubMed Scopus (600) Google Scholar (carried out by Bendifallah et al1Bendifallah S. Ballester M. Uzan C. Fauvet R. Morice P. Darai E. Nomogram to predict recurrence in patients with early- and advanced-stage mucinous and serous borderline ovarian tumors.Am J Obstet Gynecol. 2014; 211: 637.e1-637.e6Scopus (27) Google Scholar) should be repeated the within each bootstrap sample. Failure to do this and simply to assess the final model in each bootstrap sample, as done in the study by Bendifallah et al,1Bendifallah S. Ballester M. Uzan C. Fauvet R. Morice P. Darai E. Nomogram to predict recurrence in patients with early- and advanced-stage mucinous and serous borderline ovarian tumors.Am J Obstet Gynecol. 2014; 211: 637.e1-637.e6Scopus (27) Google Scholar is flawed and does not constitute a valid assessment of internal validity.3Castaldi P.J. Dahabreh I.J. Ioannidis J.P. An empirical assessment of validation practices for molecular classifiers.Brief Bioinform. 2011; 12: 189-202Crossref PubMed Scopus (57) Google Scholar Finally, the authors correctly concluded that external validation is required before the nomogram is considered in routine practice. Unfortunately, the authors have crucially failed to present the model in sufficient detail to allow external validation by independent investigators. A nomogram is merely a graphic presentation of the underlying logistic regression model to aid routine use. External validation of the model would require the actual regression model, including all regression coefficients and the intercept, to be reported. Given the sample size requirements of external validation studies,4Vergouwe Y. Steyerberg E.W. Eijkemans M.J.C. Habbema J.D.F. Substantial effective sample sizes were required for external validation studies of predictive logistic regression models.J Clin Epidemiol. 2005; 58: 475-483Abstract Full Text Full Text PDF PubMed Scopus (372) Google Scholar manually entering patient details into the nomogram to generate the predicted probabilities of recurrence is not only infeasible but also fraught with likely data entry and measurement error issues. Nomogram to predict recurrence in patients with early- and advanced-stage mucinous and serous borderline ovarian tumorsAmerican Journal of Obstetrics & GynecologyVol. 211Issue 6PreviewRecurrence prediction is a cornerstone of patient management for borderline ovarian tumors. This study aimed to develop a nomogram predicting the recurrence probability in individual patients who had received primary surgical treatment. Full-Text PDF ReplyAmerican Journal of Obstetrics & GynecologyVol. 212Issue 1PreviewWe thank Dr Gary S. Collins for his interest and his comments regarding our recent article about the development of a nomogram to predict the 5-year recurrence of borderline ovarian tumors.1 Full-Text PDF