Calculating the sample size required for developing a clinical prediction model

Toward a Modern Era in Clinical Prediction: The TRIPOD Statement for Reporting Prediction Models

Recently, the number of artificial intelligence methods used to develop clinical risk prediction models has rapidly increased. To ensure the value of clinical prediction model research, researchers must report the research content transparently, completely, and accurately. Updated Guidance for Reporting Clinical Prediction Models that Use Regression or Machine Learning Methods (TRIPOD+AI) was released in 2024 and covers a checklist of 27 major items. It aims to promote the complete reporting of global clinical prediction model research and facilitate research evaluation, model evaluation, and model implementation. This article interprets and compares aspects such as the formulation process, checklist content, applicable scenarios, and advantages of TRIPOD+AI, as well as the original Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) checklist. It also analyzes an example of predicting the depression of elderly patients using artificial intelligence methods, providing references for researchers to standardize the reporting of clinical prediction models.

IntroductionAccurate risk stratification of ED patients with infection is vital for informing priorities of investigation and treatment, disposition location, and communication to patients, families and inpatient specialists. Risk stratification systems for ED patients with infection exist in the form of ‘sepsis syndromes’ classifications and clinical prediction rules derived through mathematical modelling. However most studies designed to validate these risk stratification tools have examined patient cohorts assembled using discharge coding, critical care admission or administrative databases, or re-analyse data from studies conducted for another purpose. These factors can contribute to multiple types of bias and questionable applicability.The aims of this thesis are to demonstrate that (1) with foreknowledge of contributors to bias, a quality Australian ED dataset comprising patients with a wide spectrum of disease severity can be compiled, and 2) validations of risk stratification and clinical prediction models with these data can provide valuable insights relevant to practising ED physicians and researchers.MethodsConsecutive ED patients with infection admitted to a tertiary metropolitan hospital were enrolled in a prospective observational database. Concurrence between ED and admitting inpatient clinicians that infection was the most likely cause for admission was the primary inclusion criterion. Detailed data were recorded regarding suspected source, physiology and treatment in the ED, investigations, co-morbidities, admission location and length of stay. Mortality outcomes were sourced from a national database. Data enabled classification according to sepsis syndromes and established clinical outcome prediction models. Chapter 2 details methods as published.ResultsData were collected over 162 weeks. The study cohort comprised 9719 admissions with overall 30-day mortality 3.7%. Four papers, comprising the basis of chapters 3-6 of this thesis were published, each examining an aspect of risk stratification of ED patients admitted with infection.Sepsis syndromes: Chapter 3 examined risk stratification using ‘sepsis syndromes’ (infection without SIRS, sepsis, severe sepsis and septic shock). A 2016 reclassification examining large administrative databases advocated abandoning SIRS, proposed sequential organ function assessment (SOFA) based criteria to determine organ dysfunction, and conceived ‘q’(quick) SOFA to screen for patients with sepsis outside ICU. The paper in this chapter reported SIRS was associated with increased risk of organ dysfunction (RR 3.5) and mortality in patients without organ dysfunction (OR 3.2). SIRS and qSOFA displayed equivalent discrimination for organ dysfunction (AUROCs 0.72 vs 0.73), but SIRS provided greater sensitivity at determined operating points (72.3% vs 29.2%). Substantial variation was revealed in mortality associated with SOFA-determined dysfunction in various organ systems. A hybrid system of classification was proposed, consisting infection without SIRS, sepsis (infection with SIRS), severe sepsis (infection with organ dysfunction) and septic shock (infection with cardiovascular dysfunction).Septic shock is the subject of chapter 4, with stark contrast demonstrated between consecutive, unselected patients with this condition in the study database and cohorts enrolled in recent RCTs recruiting ED patients with septic shock. Increasing severity of illness and mortality was demonstrated for patients satisfying lactate, hypotension, and both diagnostic criteria for septic shock (mortality 14.8%, 21.3% and 27.5% respectively). Most patients with septic shock (62.7%) were not admitted to ICU, and mortality for patents admitted to ICU was lower than for patients treated on wards (12.1% vs 20.8%).Severity scores: Established clinical prediction models [MEDS, SOFA, APACHE II, SAPS II and a new ‘Severe Sepsis Score’ were validated in chapter 5, most for the first time in Australian ED patients. Spectrum bias was explored through repeated analysis in varied patient groups. MEDS showed optimal performance (AUROC 0.92), however some MEDS variables were compromised by subjective interpretation and information bias. Older scores such as APACHE II and SAPS II discriminated well (AUROC for both 0.90), but displayed poor calibration, consistently overestimating mortality.Community-acquired pneumonia (CAP). Several CAP clinical prediction models have progressed to later developmental stages including impact assessment and incorporation into guidelines. Patient disposition location is informed through prediction model stratification in several national guidelines. The final paper (chapter 6) assessed performance of establishedCAP scores, some for the first time in Australian patients. Newer scores such as SMARTCOP, CURXO and IDSA/ATS 2007 minor criteria showed higher discrimination (AUROCs 0.84-0.87) than older scores (0.70 for both PSI and CURB65). Performance of low scores was assessed for prediction of brief admission (≤ 48 hours), potentially to an ED short stay unit. No score performed sufficiently to justify this indication (AUROCs 0.64-0.74).ConclusionsThrough analysis of detailed prospective data from consecutive ED patients admitted with infection of all severities, new perspectives have emerged to challenge established constructs derived from convenience or selective sources such as administrative or RCT data. Examples include prognostic import of SIRS and insensitivity of qSOFA in the ED, and interactions between hyperlactataemia and hypotension in septic shock. Performance of severity scores was shown to be influenced by cohort selection, and endpoints. Appraisal of research examining risk stratification tools should take account of representativeness of the study cohort.

CORR Synthesis: When Should We Be Skeptical of Clinical Prediction Models?

To understand patient perceptions of specific applications of predictive models in health care. Original, cross-sectional national survey. We conducted a national online survey of US adults with the National Opinion Research Center from November to December 2021. Measures of internal consistency were used to identify how patients differentiate between clinical and administrative predictive models. Multivariable logistic regressions were used to identify relationships between comfort with various types of predictive models and patient demographics, perceptions of privacy protections, and experiences in the health caresystem. A total of 1541 respondents completed the survey. After excluding observations with missing data for the variables of interest, the final analytic sample was 1488. We found that patients differentiate between clinical and administrative predictive models. Comfort with prediction of bill payment and missed appointments was especially low (21.6% and 36.6%, respectively). Comfort was higher with clinical predictive models, such as predicting stroke in an emergency (55.8%). Experiences of discrimination were significant negative predictors of comfort with administrative predictive models. Health system transparency around privacy policies was a significant positive predictor of comfort with both clinical and administrative predictivemodels. Patients are more comfortable with clinical applications of predictive models than administrative ones. Privacy protections and transparency about how health care systems protect patient data may facilitate patient comfort with these technologies. However, larger inequities and negative experiences in health care remain important for how patients perceive administrative applications ofprediction.

Cardiac surgery-associated acute kidney injury (CSA-AKI) is a prevalent complication with poor outcomes, and its early prediction remains a challenging task. Currently available biomarkers for acute kidney injury (AKI) include serum cystatin C (sCysC) and urinary N-acetyl-β-D-glucosaminidase (uNAG). Widely used biomarkers for assessing cardiac function and injury are N-terminal pro B-type natriuretic peptide (NT-proBNP) and cardiac troponin I (cTnI). In light of this, our study aimed to evaluate the effectiveness of these four biomarkers in predicting CSA-AKI. This prospective observational study enrolled adult patients who had undergone cardiac surgery. The clinical prediction model for CSA-AKI was developed using the least absolute shrinkage and selection operator (LASSO) regression method. The model's performance was assessed using the area under the curve of the receiver operating characteristic (ROC-AUC), decision curve analysis (DCA), and calibration curves. Furthermore, a separate validation cohort was constructed to externally validate the prediction model. Additionally, a risk nomogram was created to facilitate risk assessment and prediction. In the modeling cohort consisting of 689 patients and the validation cohort consisting of 313 patients, the total incidence of CSA-AKI was 33.4%. The LASSO regression identified several predictors, including age, history of hypertension, baseline serum creatinine (sCr), coronary artery bypass grafting combined with valve surgery, cardiopulmonary bypass duration, preoperative albumin, hemoglobin, postoperative NT-proBNP, cTnI, sCysC, and uNAG. The constructed clinical prediction model demonstrated robust performance, with a ROC-AUC of 0.830 (0.800-0.860) in the modeling cohort and 0.840 (0.790-0.880) in the validation cohort. Furthermore, both calibration and DCA indicated good model fit and clinical benefit. This study demonstrates that incorporating the immediately postoperative renal biomarkers, sCysC and uNAG, along with the cardiac biomarkers, NT-proBNP and cTnI, into a clinical early prediction model can significantly enhance the accuracy of predicting CSA-AKI. These findings suggest that a comprehensive approach combining both renal and cardiac biomarkers holds promise for improving the early detection and prediction of CSA-AKI.

Assessing opioid overdose risk: a review of clinical prediction models utilizing patient-level data

Prognosis is a venerable component of medical knowledge introduced by Hippocrates (460-377 BC). This educational review presents a contemporary evidence-based approach for how to incorporate clinical risk prediction models in modern psychiatry. The article is organized around key methodological themes most relevant for the science of prognosis in psychiatry. Within each theme, the article highlights key challenges and makes pragmatic recommendations to improve scientific understanding of prognosis in psychiatry. The initial step to building clinical risk prediction models that can affect psychiatric care involves designing the model: preparation of the protocol and definition of the outcomes and of the statistical methods (theme 1). Further initial steps involve carefully selecting the predictors, preparing the data, and developing the model in these data. A subsequent step is the validation of the model to accurately test its generalizability (theme 2). The next consideration is that the accuracy of the clinical prediction model is affected by the incidence of the psychiatric condition under investigation (theme 3). Eventually, clinical prediction models need to be implemented in real-world clinical routine, and this is usually the most challenging step (theme 4). Advanced methods such as machine learning approaches can overcome some problems that undermine the previous steps (theme 5). The relevance of each of these themes to current clinical risk prediction modeling in psychiatry is discussed and recommendations are given. Together, these perspectives intend to contribute to an integrative, evidence-based science of prognosis in psychiatry. By focusing on the outcome of the individuals, rather than on the disease, clinical risk prediction modeling can become the cornerstone for a scientific and personalized psychiatry.

Prediction models: stepwise development and simultaneous validation is a step back

Understanding random resampling techniques for class imbalance correction and their consequences on calibration and discrimination of clinical risk prediction models

IntroductionIn medical and surgical intensive care units, clinical risk prediction models for readmission have been developed; however, studies reporting the risks for cardiovascular intensive care unit (CVICU) readmission have been methodologically limited by small numbers of outcomes, unreported measures of calibration or discrimination, or a lack of information spanning the entire perioperative period. The purpose of this study was to derive and validate a clinical prediction model for CVICU readmission in cardiac surgical patients.MethodsA total of 10,799 patients more than or equal to 18 years in the Alberta Provincial Project for Outcomes Assessment in Coronary Heart Disease (APPROACH) registry who underwent cardiac surgery (coronary artery bypass or valvular surgery) between 2004 and 2012 and were discharged alive from the first CVICU admission were included. The full cohort was used to derive the clinical prediction model and the model was internally validated with bootstrapping. Discrimination and calibration were assessed using the AUC c index and the Hosmer-Lemeshow tests, respectively.ResultsA total of 479 (4.4%) patients required CVICU readmission. The mean CVICU length of stay (19.9 versus 3.3 days, P <0.001) and in-hospital mortality (14.4% versus 2.2%, P <0.001) were higher among patients readmitted to the CVICU. In the derivation cohort, a total of three preoperative (age ≥70, ejection fraction, chronic lung disease), two intraoperative (single valve repair or replacement plus non-CABG surgery, multivalve repair or replacement), and seven postoperative variables (cardiac arrest, pneumonia, pleural effusion, deep sternal wound infection, leg graft harvest site infection, gastrointestinal bleed, neurologic complications) were independently associated with CVICU readmission. The clinical prediction model had robust discrimination and calibration in the derivation cohort (AUC c index = 0.799; Hosmer-Lemeshow P = 0.192). The validation point estimates and confidence intervals were similar to derivation model.ConclusionsIn a large population-based dataset incorporating a comprehensive set of perioperative variables, we have derived a clinical prediction model with excellent discrimination and calibration. This model identifies opportunities for targeted therapeutic interventions aimed at reducing CVICU readmissions in high-risk patients.

Poor ovarian response (POR) is associated with decreased clinical pregnancy rates, emphasizing the need for developing clinical prediction models. Such models can improve prognostic accuracy, personalize medical interventions, and ultimately enhance live birth rates among patients with POR. This study aims to develop and validate a prognostic model for predicting clinical pregnancy outcomes in individuals with POR undergoing in vitro fertilization/ intracytoplasmic sperm injection (IVF/ICSI) cycles. A retrospective cohort of 969 patients with POR undergoing fresh embryo transfer cycles at the Reproductive Center of Fujian Maternal and Child Health Center from January 2018 to January 2022 was included. The cohort was randomly divided into model (n = 678) and validation (n = 291) groups in a 7:3 ratio. A single-factor analysis was performed on the model group to identify variables influencing clinical pregnancy. Optimal variables were selected using LASSO regression, and a clinical prediction model was constructed using multivariate logistic regression analysis. The model's calibration and discrimination were assessed using receiver operating characteristic (ROC) and calibration curves, while the clinical utility was evaluated using decision curve analysis. Multivariate logistic regression analysis revealed that the age of the women (odds ratio [OR] 0.936, 95% confidence interval [CI] 0.898-0.976, P = 0.002), body mass index (BMI) ≤ 24 (OR 2.748, 95% CI 1.724-4.492, P < 0.001), antral follicle count (AFC) (OR 1.232, 95% CI 1.073-1.416, P = 0.003), anti-Müllerian hormone (AMH) (OR 1.67, 95% CI 1.178-2.376, P = 0.004), number of mature oocytes (OR 1.227, 95% CI 1.075-1.403, P = 0.003), number of embryos transferred (OR 1.692, 95% CI 1.132-2.545, P = 0.011), and transfer of high-quality embryos (OR 3.452, 95% CI 1.548-8.842, P = 0.005) were independent predictors of clinical pregnancy in patients with POR. According to the receiver operating characteristic (ROC) analysis, the prediction model exhibited an area under the curve (AUC) of 0.752 (0.714, 0.789) in the model group and 0.765 (0.708, 0.821) in the validation group. The clinical decision curve demonstrated that the model held maximum clinical utility in both cohorts when the threshold probability of clinical pregnancy ranged from 6-81% to 12-82%, respectively. Clinical pregnancy outcomes in patients with POR who underwent IVF/ICSI treatment were influenced by several independent factors, including the age of the women, BMI, AFC, AMH, number of mature oocytes, number of embryos transferred, and transfer of high-quality embryos. A clinical prediction model based on these factors exhibited favorable clinical predictive and applicative value. Therefore, this model can serve as a valuable tool for clinical prognosis, intervention, and facilitating personalized medical treatment.

Prognostic relevance of baseline pro- and anti-inflammatory markers in STEMI: An APEX AMI substudy

Introduction: Clinicians may find early identification of patients at risk for high cost of care during and after surgery useful, to prepare for focused management that results in optimal clinical outcome. The aim of the study was to develop a clinical prediction model to identify high and low hospital cost outcome after elective non-cardiac surgery using predictors identified from a preoperative self-assessment questionnaire. Methods: Data to develop a clinical prediction model were collected for this purpose at a private hospital in South Africa. Predictors were defined from a preoperative questionnaire. Cost of hospital admission data were received from hospital administration, which reflected the financial risk the hospital carries and which could be reasonably attributed to a patient’s individual clinical risk profile. The hospital cost excluded fees charged (by any healthcare provider), and cost of prosthesis and other consignment items that are related to the type of procedure. The cost outcome measure was described as cost per total Work Relative Value Units (Work RVUs) for the procedure, and dichotomised. Variables that were associated with the outcome during univariate analysis were subjected to a forward stepwise regression selection technique. The prediction model was evaluated for discrimination and calibration, and internally validated. Results: Data from 770 participants were used to develop the prediction model. The number of participants with the outcome of high cost were 142/770 (18.4%). The predictors included in the full prediction model were type of surgery, treatment for chronic pain with depression, and activity status. The area under the receiver operating curve (AUROC) for the prediction model was 0.83 (95% confidence interval [CI]: 0.79 to 0.86). The Hosmer–Lemeshow indicated goodness-of-fit (p = 0.967). The prediction model was internally validated using bootstrap resampling from the development cohort, with a resultant AUROC of 0.86 (95% CI: 0.82 to 0.89). Conclusion: The study describes a clinical risk prediction model developed using easily collected patient-reported variables and readily available administrative information. The prediction model should be validated and updated using a larger dataset, and used to identify patients in which cost-effective care pathways can add value.

Based on the literature and data on its clinical trials, the incidence of venous thromboembolism (VTE) in patients undergoing neurosurgery has been 3.0%~26%. We used advanced machine learning techniques and statistical methods to provide a clinical prediction model for VTE after neurosurgery. All patients (n = 5867) who underwent neurosurgery from the development and retrospective internal validation cohorts were obtained from May 2017 to April 2022 at the Department of Neurosurgery at the Sanbo Brain Hospital. The clinical and biomarker variables were divided into pre-, intra-, and postoperative. A univariate logistic regression (LR) was applied to explore the 67 candidate predictors with VTE. We used a multivariable logistic regression (MLR) to select all significant MLR variables of MLR to build the clinical risk prediction model. We used a random forest to calculate the importance of significant variables of MLR. In addition, we conducted prospective internal (n = 490) and external validation (n = 2301) for the model. Eight variables were selected for inclusion in the final clinical prediction model: D-dimer before surgery, activated partial thromboplastin time before neurosurgery, age, craniopharyngioma, duration of operation, disturbance of consciousness on the second day after surgery and high dose of mannitol, and highest D-dimer within 72 h after surgery. The area under the curve (AUC) values for the development, retrospective internal validation, and prospective internal validation cohorts were 0.78, 0.77, and 0.79, respectively. The external validation set had the highest AUC value of 0.85. This validated clinical prediction model, including eight clinical factors and biomarkers, predicted the risk of VTE following neurosurgery. Looking forward to further research exploring the standardization of clinical decision-making for primary VTE prevention based on this model.