Problems With the Development and Validation of a Prognostic Model

Abstract

To the Editor: With high quality data and appropriate statistical methods, prognostic models can hold enormous potential for identifying individuals at increased risk developing a health condition. Unfortunately, the recent study described by He et al (1.He X Xu G Liang W Nomogram for predicting time to death after withdrawal of life-sustaining treatment in patients with devastating neurological injury.Am J Transplant. 2015; 15 (et al): 2136-2142Abstract Full Text Full Text PDF PubMed Scopus (15) Google Scholar) not only fails to omit key important information as suggested the TRIPOD Initiative (2.Collins GS Reitsma JB Altman DG Moons KGM. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): The TRIPOD statement.Ann Intern Med. 2015; 162: 55-63Crossref PubMed Scopus (1217) Google Scholar) (www.tripod-statement.org), but has numerous methodological problems which deserve highlighting so that future investigators do not fall into the same trap. Sample size considerations for studies developing a prognostic model are frequently based on the rule-of-thumb of (a minimum of) 10 events per variable (EPV) (3.Steyerberg EW. Clinical prediction models: A practical approach to development, validation, and updating. Springer, New York2009Crossref Google Scholar). A small EPV will result in an overfit model, that is, a model that fits the data too well and ultimately describes noise or random error and is when too many predictors are examined in relation to the number of events. The performance of the model will then be overestimated (termed optimistic). The “event” is the smaller of the number of individuals experiencing the event or the number of individuals not experiencing the event. In the study by He et al, 123 died with 60 min and 52 did not; thus, the study had 52 “events.” The number of predictors examined was 46, leading to an EPV of 52/46 = 1.1, which is much lower than the recommended minimum of EPV = 10. The authors then proceed to evaluate the performance of the model on two separate datasets, of size 201 and 43 (the number of deaths within 60 min is disappointingly not reported) and reported rather spectacular performance. It is worth remarking that sample size considerations for studies validating a prediction model are that a minimum of 100 events and 100 nonevents are required (4.Vergouwe Y Steyerberg EW Eijkemans MJC Habbema JDF. Substantial effective sample sizes were required for external validation studies of predictive logistic regression models.J Clin Epidemiol. 2005; 58: 475-483Abstract Full Text Full Text PDF PubMed Scopus (398) Google Scholar); values most certainly not achieved in the study by He et al. The final, and arguably, the most important reporting issue affecting reproducibility and implementation relates to the presentation of the model; the authors produced a nomogram. A nomogram is not a prognostic model but merely a graphical presentation of the underlying regression model. For other independent investigators wishing to evaluate (i.e. validate) on other data, it is absolutely vital that the underlying model, namely all regression coefficients plus the baseline survival at 30, 60, 120, and 240 min (which the authors have not done) are clearly reported. In the absence of the full model, independent validation of the model by independent investigators is not possible. We recommend the authors and other investigators contemplating developing or validating a prediction model to consult to the Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD) Statement (2.Collins GS Reitsma JB Altman DG Moons KGM. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): The TRIPOD statement.Ann Intern Med. 2015; 162: 55-63Crossref PubMed Scopus (1217) Google Scholar). In addition to providing guidance on key information to report when describing a prognostic model study, the accompanying Explanation & Elaboration article also highlights many methodological considerations when developing or validation a clinical prediction model (5.Moons KGM Altman DG Reitsma JB Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): Explanation and elaboration.Ann Intern Med. 2015; 162 (et al): W1-W73Crossref PubMed Scopus (1167) Google Scholar). The authors of this manuscript have no conflicts of interest to disclose as described by the American Journal of Transplantation.