Internal-external Cross-validation Research Articles

#2801 Which patients may benefit most from hemodiafiltration compared with hemodialysis: prediction of the individualized treatment effect

Abstract Background and Aims Haemodiafiltration (HDF) with an appropriate dosing reduces death from any cause based on an individual participant data meta-analysis and aggregated data meta-analyses of available randomized trials. No differences in treatment effect are reported for specific patient populations when defined on a single characteristic in subgroup analyses. Yet, it is unclear to what extent individual patients, characterized by a combination of clinical characteristics, may benefit from HDF. This study aimed to develop and internally validate a treatment effect prediction model to determine which individual patients would benefit most from well dosed HDF, compared with haemodialysis (HD), in terms of survival. Method Individual participant data from five randomized controlled trials (CONTRAST, ESHOL, Turkish HDF study, French HDF study, CONVINCE) comparing HDF with HD on all-cause mortality were used to derive a Royston-Parmar model for exploring the prediction of absolute treatment effect of HDF based on pre-specified patient and disease characteristics, notably age, sex, body mass index, diabetes mellitus, history of cardiovascular disease, creatinine levels, and c-reactive protein levels. Internal validation of the model was performed using internal-external cross validation. Results Among 4153 participants, with a median follow-up of 30 months (Q1-Q3: 24-36), death from any cause occurred in 558 patients (27.2%). The median predicted survival benefit of HDF compared with HD was 6.9 (Q1–Q3: 5.6–9.0) months, with a range of 2 to 42 months. Patients who were predicted to benefit most from HDF were younger, less likely to have diabetes or a cardiovascular history and had higher serum creatinine levels. Internal-external cross validation showed adequate discrimination and calibration. Conclusion All-cause mortality is reduced by HDF compared with HD in ESKD patients. Yet, the absolute survival benefit seems to considerably differ between individual patients. Our results suggest that the effects of HDF on absolute survival can be predicted accurately using a combination of readily available patient and disease characteristics. This prediction model approach exemplifies the potential of prediction algorithms in supporting clinical decision-making for the choice of treatment options. Validation and potential updates to the presented algorithm are necessary, along with a preferable randomized evaluation of its clinical impact before widespread implementation.

Predicting the risk of pancreatic cancer in adults with new-onset diabetes: development and internal-external validation of a clinical risk prediction model.

The National Institute for Health and Care Excellence (NICE) recommends that people aged 60+ years with newly diagnosed diabetes and weight loss undergo abdominal imaging to assess for pancreatic cancer. More nuanced stratification could lead to enrichment of these referral pathways. Population-based cohort study of adults aged 30-85 years at type 2 diabetes diagnosis (2010-2021) using the QResearch primary care database in England linked to secondary care data, the national cancer registry and mortality registers. Clinical prediction models were developed to estimate risks of pancreatic cancer diagnosis within 2 years and evaluated using internal-external cross-validation. Seven hundred and sixty-seven of 253,766 individuals were diagnosed with pancreatic cancer within 2 years. Models included age, sex, BMI, prior venous thromboembolism, digoxin prescription, HbA1c, ALT, creatinine, haemoglobin, platelet count; and the presence of abdominal pain, weight loss, jaundice, heartburn, indigestion or nausea (previous 6 months). The Cox model had the highest discrimination (Harrell's C-index 0.802 (95% CI: 0.797-0.817)), the highest clinical utility, and was well calibrated. The model's highest 1% of predicted risks captured 12.51% of pancreatic cancer cases. NICE guidance had 3.95% sensitivity. A new prediction model could have clinical utility in identifying individuals with recent onset diabetes suitable for fast-track abdominal imaging.

Development and Validation of Prediction Models for the 5-year Risk of Type 2 Diabetes in a Japanese Population: Japan Public Health Center-based Prospective (JPHC) Diabetes Study.

This study aimed to develop models to predict the 5-year incidence of type 2 diabetes mellitus (T2DM) in a Japanese population and validate them externally in an independent Japanese population. Data from 10,986 participants (aged 46-75 years) in the development cohort of the Japan Public Health Center-based Prospective Diabetes Study and 11,345 participants (aged 46-75 years) in the validation cohort of the Japan Epidemiology Collaboration on Occupational Health Study were used to develop and validate the risk scores in logistic regression models. We considered non-invasive (sex, body mass index, family history of diabetes mellitus, and diastolic blood pressure) and invasive (glycated hemoglobin [HbA1c] and fasting plasma glucose [FPG]) predictors to predict the 5-year probability of incident diabetes. The area under the receiver operating characteristic curve was 0.643 for the non-invasive risk model, 0.786 for the invasive risk model with HbA1c but not FPG, and 0.845 for the invasive risk model with HbA1c and FPG. The optimism for the performance of all models was small by internal validation. In the internal-external cross-validation, these models tended to show similar discriminative ability across different areas. The discriminative ability of each model was confirmed using external validation datasets. The invasive risk model with only HbA1c was well-calibrated in the validation cohort. Our invasive risk models are expected to discriminate between high- and low-risk individuals with T2DM in a Japanese population.

Predicting effects of a digital stress intervention for patients with depressive symptoms: Development and validation of meta-analytic prognostic models using individual participant data.

Digital stress interventions could be helpful as an "indirect" treatment for depression, but it remains unclear for whom this is a viable option. In this study, we developed models predicting individualized benefits of a digital stress intervention on depressive symptoms at 6-month follow-up. Data of N = 1,525 patients with depressive symptoms (Center for Epidemiological Studies' Depression Scale, CES-D ≥ 16) from k = 6 randomized trials (digital stress intervention vs. waitlist) were collected. Prognostic models were developed using multilevel least absolute shrinkage and selection operator and boosting algorithms, and were validated using bootstrap bias correction and internal-external cross-validation. Subsequently, expected effects among those with and without a treatment recommendation were estimated based on clinically derived treatment assignment cut points. Performances ranged from R² = 21.0%-23.4%, decreasing only slightly after model optimism correction (R² = 17.0%-19.6%). Predictions were greatly improved by including an interim assessment of depressive symptoms (optimism-corrected R2 = 32.6%-35.6%). Using a minimally important difference of d = -0.24 as assignment cut point, approximately 84.6%-93.3% of patients are helped by this type of intervention, while the remaining 6.7%-15.4% would experience clinically negligible benefits (δ^ = -0.02 to -0.19). Using reliable change as cut point, a smaller subset (39.3%-46.2%) with substantial expected benefits (δ^ = -0.68) receives a treatment recommendation. Meta-analytic prognostic models applied to individual participant data can be used to predict differential benefits of a digital stress intervention as an indirect treatment for depression. While most patients seem to benefit, the developed models could be helpful as a screening tool to identify those for whom a more intensive depression treatment is needed. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Development and external validation of the eFalls tool: a multivariable prediction model for the risk of ED attendance or hospitalisation with a fall or fracture in older adults.

Falls are common in older adults and can devastate personal independence through injury such as fracture and fear of future falls. Methods to identify people for falls prevention interventions are currently limited, with high risks of bias in published prediction models. We have developed and externally validated the eFalls prediction model using routinely collected primary care electronic health records (EHR) to predict risk of emergency department attendance/hospitalisation with fall or fracture within 1 year. Data comprised two independent, retrospective cohorts of adults aged ≥65 years: the population of Wales, from the Secure Anonymised Information Linkage Databank (model development); the population of Bradford and Airedale, England, from Connected Bradford (external validation). Predictors included electronic frailty index components, supplemented with variables informed by literature reviews and clinical expertise. Fall/fracture risk was modelled using multivariable logistic regression with a Least Absolute Shrinkage and Selection Operator penalty. Predictive performance was assessed through calibration, discrimination and clinical utility. Apparent, internal-external cross-validation and external validation performance were assessed across general practices and in clinically relevant subgroups. The model's discrimination performance (c-statistic) was 0.72 (95% confidence interval, CI: 0.68 to 0.76) on internal-external cross-validation and 0.82 (95% CI: 0.80 to 0.83) on external validation. Calibration was variable across practices, with some over-prediction in the validation population (calibration-in-the-large, -0.87; 95% CI: -0.96 to -0.78). Clinical utility on external validation was improved after recalibration. The eFalls prediction model shows good performance and could support proactive stratification for falls prevention services if appropriately embedded into primary care EHR systems.

Updating M6 pregnancy of unknown location risk-prediction model including evaluation of clinical factors.

Ectopic pregnancy (EP) is a major high-risk outcome following a pregnancy of unknown location (PUL) classification. Biochemical markers are used to triage PUL as high vs low risk to guide appropriate follow-up. The M6 model is currently the best risk-prediction model. We aimed to update the M6 model and evaluate whether performance can be improved by including clinical factors. This prospective cohort study recruited consecutive PUL between January 2015 and January 2017 at eight units (Phase 1), with two centers continuing recruitment between January 2017 and March 2021 (Phase 2). Serum samples were collected routinely and sent for β-human chorionic gonadotropin (β-hCG) and progesterone measurement. Clinical factors recorded were maternal age, pain score, bleeding score and history of EP. Based on transvaginal ultrasonography and/or biochemical confirmation during follow-up, PUL were classified subsequently as failed PUL (FPUL), intrauterine pregnancy (IUP) or EP (including persistent PUL (PPUL)). The M6 models with (M6P ) and without (M6NP ) progesterone were refitted and extended with clinical factors. Model validation was performed using internal-external cross-validation (IECV) (Phase 1) and temporal external validation (EV) (Phase 2). Missing values were handled using multiple imputation. Overall, 5473 PUL were recruited over both phases. A total of 709 PUL were excluded because maternal age was < 16 years or initial β-hCG was ≤ 25 IU/L, leaving 4764 (87%) PUL for analysis (2894 in Phase 1 and 1870 in Phase 2). For the refitted M6P model, the area under the receiver-operating-characteristics curve (AUC) for EP/PPUL vs IUP/FPUL was 0.89 for IECV and 0.84-0.88 for EV, with respective sensitivities of 94% and 92-93%. For the refitted M6NP model, the AUCs were 0.85 for IECV and 0.82-0.86 for EV, with respective sensitivities of 92% and 93-94%. Calibration performance was good overall, but with heterogeneity between centers. Net Benefit confirmed clinical utility. The change in AUC when M6P was extended to include maternal age, bleeding score and history of EP was between -0.02 and 0.01, depending on center and phase. The corresponding change in AUC when M6NP was extended was between -0.01 and 0.03. At the 5% threshold to define high risk of EP/PPUL, extending M6P altered sensitivity by -0.02 to -0.01, specificity by0.03 to 0.04 and Net Benefit by -0.005 to 0.006. Extending M6NP altered sensitivity by -0.03 to -0.01, specificity by 0.05 to 0.07 and Net Benefit by -0.005 to 0.006. The updated M6 model offers accurate diagnostic performance, with excellent sensitivity for EP. Adding clinical factors to the model improved performance in some centers, especially when progesterone levels were not suitable or unavailable. © 2023 The Authors. Ultrasound in Obstetrics & Gynecology published by John Wiley & Sons Ltd on behalf of International Society of Ultrasound in Obstetrics and Gynecology.

Machine Learning Models for Predicting Disability and Pain Following Lumbar Disc Herniation Surgery

Lumber disc herniation surgery can reduce pain and disability. However, a sizable minority of individuals experience minimal benefit, necessitating the development of accurate prediction models. To develop and validate prediction models for disability and pain 12 months after lumbar disc herniation surgery. A prospective, multicenter, registry-based prognostic study was conducted on a cohort of individuals undergoing lumbar disc herniation surgery from January 1, 2007, to May 31, 2021. Patients in the Norwegian Registry for Spine Surgery from all public and private hospitals in Norway performing spine surgery were included. Data analysis was performed from January to June 2023. Microdiscectomy or open discectomy. Treatment success at 12 months, defined as improvement in Oswestry Disability Index (ODI) of 22 points or more; Numeric Rating Scale (NRS) back pain improvement of 2 or more points, and NRS leg pain improvement of 4 or more points. Machine learning models were trained for model development and internal-external cross-validation applied over geographic regions to validate the models. Model performance was assessed through discrimination (C statistic) and calibration (slope and intercept). Analysis included 22 707 surgical cases (21 161 patients) (ODI model) (mean [SD] age, 47.0 [14.0] years; 12 952 [57.0%] males). Treatment nonsuccess was experienced by 33% (ODI), 27% (NRS back pain), and 31% (NRS leg pain) of the patients. In internal-external cross-validation, the selected machine learning models showed consistent discrimination and calibration across all 5 regions. The C statistic ranged from 0.81 to 0.84 (pooled random-effects meta-analysis estimate, 0.82; 95% CI, 0.81-0.84) for the ODI model. Calibration slopes (point estimates, 0.94-1.03; pooled estimate, 0.99; 95% CI, 0.93-1.06) and calibration intercepts (point estimates, -0.05 to 0.11; pooled estimate, 0.01; 95% CI, -0.07 to 0.10) were also consistent across regions. For NRS back pain, the C statistic ranged from 0.75 to 0.80 (pooled estimate, 0.77; 95% CI, 0.75-0.79); for NRS leg pain, the C statistic ranged from 0.74 to 0.77 (pooled estimate, 0.75; 95% CI, 0.74-0.76). Only minor heterogeneity was found in calibration slopes and intercepts. The findings of this study suggest that the models developed can inform patients and clinicians about individual prognosis and aid in surgical decision-making.

Dynamic Prediction of Advanced Colorectal Neoplasia in Inflammatory Bowel Disease

Colonoscopic surveillance is recommended in patients with colonic inflammatory bowel disease (IBD) given their increased risk of colorectal cancer (CRC). We aimed to develop and validate a dynamic prediction model for the occurrence of advanced colorectal neoplasia (aCRN, including high-grade dysplasia and CRC) in IBD. We pooled data from six existing cohort studies from Canada, the Netherlands, the UK, and the USA. Patients with IBD and an indication for CRC surveillance were included if they underwent at least one follow-up procedure. Exclusion criteria included prior aCRN, prior colectomy, or an unclear indication for surveillance. Predictor variables were selected based on literature. A dynamic prediction model was developed using a landmarking approach based on Cox proportional hazard modelling. Model performance was assessed with Harrell's concordance-statistic (discrimination) and by calibration curves. Generalizability across surveillance cohorts was evaluated by internal-external cross-validation. The surveillance cohorts comprised 3,731 patients, enrolled and followed-up in the time period 1973-2021, with a median follow-up of 5.7 years (26,336 patient-years of follow-up); 146 individuals were diagnosed with aCRN. The model contained eight predictors, with a cross-validation median c-statistic of 0.74 and 0.75 for a 5- and 10-year prediction window, respectively. Calibration plots showed good calibration. Internal-external cross-validation results showed medium discrimination and reasonable to good calibration. The new prediction model showed good discrimination and calibration, however, generalizability results varied. Future research should focus on formal external validation and relate predicted aCRN risks to surveillance intervals prior to clinical application.

A prediction model for response to immune checkpoint inhibition in advanced melanoma.

Predicting who will benefit from treatment with immune checkpoint inhibition (ICI) in patients with advanced melanoma is challenging. We developed a multivariable prediction model for response to ICI, using routinely available clinical data including primary melanoma characteristics. We used a population-based cohort of 3525 patients with advanced cutaneous melanoma treated with anti-PD-1-based therapy. Our prediction model for predicting response within 6 months after ICI initiation was internally validated with bootstrap resampling. Performance evaluation included calibration, discrimination and internal-external cross-validation. Included patients received anti-PD-1 monotherapy (n = 2366) or ipilimumab plus nivolumab (n = 1159) in any treatment line. The model included serum lactate dehydrogenase, World Health Organization performance score, type and line of ICI, disease stage and time to first distant recurrence-all at start of ICI-, and location and type of primary melanoma, the presence of satellites and/or in-transit metastases at primary diagnosis and sex. The over-optimism adjusted area under the receiver operating characteristic was 0.66 (95% CI: 0.64-0.66). The range of predicted response probabilities was 7%-81%. Based on these probabilities, patients were categorized into quartiles. Compared to the lowest response quartile, patients in the highest quartile had a significantly longer median progression-free survival (20.0 vs 2.8 months; P < .001) and median overall survival (62.0 vs 8.0 months; P < .001). Our prediction model, based on routinely available clinical variables and primary melanoma characteristics, predicts response to ICI in patients with advanced melanoma and discriminates well between treated patients with a very good and very poor prognosis.

Predicting adverse outcomes in adults with a community-acquired lower respiratory tract infection: a protocol for the development and validation of two prediction models for (i) all-cause hospitalisation and mortality and (ii) cardiovascular outcomes

BackgroundCommunity-acquired lower respiratory tract infections (LRTI) are common in primary care and patients at particular risk of adverse outcomes, e.g., hospitalisation and mortality, are challenging to identify. LRTIs are also linked to an increased incidence of cardiovascular diseases (CVD) following the initial infection, whereas concurrent CVD might negatively impact overall prognosis in LRTI patients. Accurate risk prediction of adverse outcomes in LRTI patients, while considering the interplay with CVD, can aid general practitioners (GP) in the clinical decision-making process, and may allow for early detection of deterioration. This paper therefore presents the design of the development and external validation of two models for predicting individual risk of all-cause hospitalisation or mortality (model 1) and short-term incidence of CVD (model 2) in adults presenting to primary care with LRTI.MethodsBoth models will be developed using linked routine electronic health records (EHR) data from Dutch primary and secondary care, and the mortality registry. Adults aged ≥ 40 years with a GP-diagnosis of LRTI between 2016 and 2019 are eligible for inclusion. Relevant patient demographics, medical history, medication use, presenting signs and symptoms, and vital and laboratory measurements will be considered as candidate predictors. Outcomes of interest include 30-day all-cause hospitalisation or mortality (model 1) and 90-day CVD (model 2). Multivariable elastic net regression techniques will be used for model development. During the modelling process, the incremental predictive value of CVD for hospitalisation or all-cause mortality (model 1) will also be assessed. The models will be validated through internal-external cross-validation and external validation in an equivalent cohort of primary care LRTI patients.DiscussionImplementation of currently available prediction models for primary care LRTI patients is hampered by limited assessment of model performance. While considering the role of CVD in LRTI prognosis, we aim to develop and externally validate two models that predict clinically relevant outcomes to aid GPs in clinical decision-making. Challenges that we anticipate include the possibility of low event rates and common problems related to the use of EHR data, such as candidate predictor measurement and missingness, how best to retrieve information from free text fields, and potential misclassification of outcome events.

Development and internal-external validation of a prediction model for premature ventricular contraction unresponsive to the medical treatment

Abstract Background Ventricular premature contractions (PVCs) are the most common ventricular arrhythmias. However, to date, no risk stratification tool exists to assess response to medical therapy in patients with frequent PVCs (&amp;gt;5% per 24 hours). We aimed to develop and validate a clinical prediction model evaluating response to medical therapy in frequent PVCs. Methods We conducted a retrospective cohort study of patients with frequent PVCs who were considered for medical treatment. The study outcome was unresponsiveness to the medical treatment (24-hour PVC count not diminishing by at least 80%) in patients with frequent PVCs at 3-6 months follow-up. Potential predictors were, age (years), PVC burden (%), LVEF % (&amp;lt;55, ³55), PVC QRS width (msec), mean heart rate (beat/min), sinus beat QTc (msec), PVC coupling interval, gender, presence of multifocal PVC and Non-sustained VT. Binary logistic regression analyses were performed to develop and internally validate the model. Finally, we used internal-external cross validation using leave-one-out center. Results 1644 patients were included in the study (mean age 52.2±13.5 years and 56.3% male). The frequency of unresponsiveness to the medical treatment in patients with frequent PVC was 31.2% (n=513). In the model, PVC burden (%), LVEF% (&amp;lt;55, ³55) and PVC QRS width were found to be the three strongest predictors. The apparent and internal validation discriminations of the model (C-statistics in internal validation 0.910) were quite satisfactory. Model discriminations and calibration metrics with internal-external cross validation were similar to the apparent model and were deemed acceptable (https://demonoreflow.shinyapps.io/dynnomapp/). Conclusions We developed and validated a model that accurately predicts unresponsiveness to medical treatment in patients with frequent PVC. Our model, once externally validated, has the potential to facilitate management decisions by providing individualized risk estimates in patients with frequent PVC for whom ablation may be suggested as a first line treatment modality.Figure-1Figure-2

Development and validation of a clinical prediction model for the diagnostic management of acute pulmonary embolism: an individual participant data meta-analysis

Abstract Background/Introduction Risk stratification is used for decisions about imaging in patients with clinically suspected acute pulmonary embolism (PE). Purpose To develop a clinical prediction model that provides an individualized, accurate probability estimate for acute pulmonary embolism (PE) based on readily available clinical items and D-dimer concentrations. Methods An individual participant data meta-analysis was performed based on sixteen studies with data from 28,305 adult patients with clinically suspected PE from various clinical settings. A multilevel logistic regression model was built and validated including ten a priori defined objective candidate predictors to predict PE at baseline or during follow-up of 30-90 days. Multiple imputation was used for missing data. Backward elimination was performed with a P-value &amp;lt; 0.10. Discrimination (c-statistic with 95% CI and prediction intervals [PI]) and calibration (outcome:expected [O:E] ratio and calibration plot) were evaluated based on internal-external cross-validation. The accuracy of the model was compared to algorithms based on the Wells score and D-dimer testing. Results The final model included age (in years), sex, previous VTE, recent surgery/immobilization, hemoptysis, cancer, clinical signs of DVT, inpatient status, D-dimer (in μg/L), and an interaction term between age and D-dimer. The pooled c-statistic was 0.87 (95%-CI, 0.85-0.89; 95%-PI, 0.77-0.93) and overall calibration was very good (pooled O:E ratio, 0.99; 95% CI, 0.87-1.14; 95% PI, 0.55-1.79). In the lower range of probabilities, the model slightly overestimated VTE probability. Discrimination of the current model in the validation data sets was better than that of the Wells score combined with a D-dimer threshold based on age (c- statistic 0.73; 95%-CI, 0.70-0.75) or structured clinical pretest probability (c-statistic 0.79; 95%-CI, 0.76- 0.81). Conclusion(s) The present model provides an absolute, individualized probability of PE presence in a broad population of patients with suspected PE, with very good discrimination and calibration. Its clinical utility needs to be evaluated in a prospective management or impact study.Calibration plot of the new model

Prognosis and prediction of antibiotic benefit in adults with clinically diagnosed acute rhinosinusitis: an individual participant data meta-analysis

BackgroundA previous individual participant data meta-analysis (IPD-MA) of antibiotics for adults with clinically diagnosed acute rhinosinusitis (ARS) showed a marginal overall effect of antibiotics, but was unable to identify patients that are most likely to benefit from antibiotics when applying conventional (i.e. univariable or one-variable-at-a-time) subgroup analysis. We updated the systematic review and investigated whether multivariable prediction of patient-level prognosis and antibiotic treatment effect may lead to more tailored treatment assignment in adults presenting to primary care with ARS.MethodsAn IPD-MA of nine double-blind placebo-controlled trials of antibiotic treatment (n=2539) was conducted, with the probability of being cured at 8–15 days as the primary outcome. A logistic mixed effects model was developed to predict the probability of being cured based on demographic characteristics, signs and symptoms, and antibiotic treatment assignment. Predictive performance was quantified based on internal-external cross-validation in terms of calibration and discrimination performance, overall model fit, and the accuracy of individual predictions.ResultsResults indicate that the prognosis with respect to risk of cure could not be reliably predicted (c-statistic 0.58 and Brier score 0.24). Similarly, patient-level treatment effect predictions did not reliably distinguish between those that did and did not benefit from antibiotics (c-for-benefit 0.50).ConclusionsIn conclusion, multivariable prediction based on patient demographics and common signs and symptoms did not reliably predict the patient-level probability of cure and antibiotic effect in this IPD-MA. Therefore, these characteristics cannot be expected to reliably distinguish those that do and do not benefit from antibiotics in adults presenting to primary care with ARS.

Development and internal validation of a clinical prediction model for serious complications after emergency laparotomy

PurposeEmergency laparotomy (EL) is a common operation with high risk for postoperative complications, thereby requiring accurate risk stratification to manage vulnerable patients optimally. We developed and internally validated a predictive model of serious complications after EL.MethodsData for eleven carefully selected candidate predictors of 30-day postoperative complications (Clavien-Dindo grade > = 3) were extracted from the HELAS cohort of EL patients in 11 centres in Greece and Cyprus. Logistic regression with Least Absolute Shrinkage and Selection Operator (LASSO) was applied for model development. Discrimination and calibration measures were estimated and clinical utility was explored with decision curve analysis (DCA). Reproducibility and heterogeneity were examined with Bootstrap-based internal validation and Internal–External Cross-Validation. The American College of Surgeons National Surgical Quality Improvement Program’s (ACS-NSQIP) model was applied to the same cohort to establish a benchmark for the new model.ResultsFrom data on 633 eligible patients (175 complication events), the SErious complications After Laparotomy (SEAL) model was developed with 6 predictors (preoperative albumin, blood urea nitrogen, American Society of Anaesthesiology score, sepsis or septic shock, dependent functional status, and ascites). SEAL had good discriminative ability (optimism-corrected c-statistic: 0.80, 95% confidence interval [CI] 0.79–0.81), calibration (optimism-corrected calibration slope: 1.01, 95% CI 0.99–1.03) and overall fit (scaled Brier score: 25.1%, 95% CI 24.1–26.1%). SEAL compared favourably with ACS-NSQIP in all metrics, including DCA across multiple risk thresholds.ConclusionSEAL is a simple and promising model for individualized risk predictions of serious complications after EL. Future external validations should appraise SEAL’s transportability across diverse settings.

Predicting procedure duration of colorectal endoscopic submucosal dissection at Western endoscopy centers

Abstract Background and study aims Overcoming logistical obstacles for the implementation of colorectal endoscopic submucosal dissection (ESD) requires accurate prediction of procedure times. We aimed to evaluate existing and new prediction models for ESD duration. Patients and methods Records of all consecutive patients who underwent single, non-hybrid colorectal ESDs before 2020 at three Dutch centers were reviewed. The performance of an Eastern prediction model [GIE 2021;94(1):133–144] was assessed in the Dutch cohort. A prediction model for procedure duration was built using multivariable linear regression. The model’s performance was validated using internal validation by bootstrap resampling, internal-external cross-validation and external validation in an independent Swedish ESD cohort. Results A total of 435 colorectal ESDs were analyzed (92% en bloc resections, mean duration 139 minutes, mean tumor size 39 mm). The performance of current unstandardized time scheduling practice was suboptimal (explained variance: R2=27%). We successfully validated the Eastern prediction model for colorectal ESD duration &lt;60 minutes (c-statistic 0.70, 95% CI 0.62–0.77), but this model was limited due to dichotomization of the outcome and a relatively low frequency (14%) of ESDs completed &lt;60 minutes in the Dutch centers. The model was more useful with a dichotomization cut-off of 120 minutes (c-statistic: 0.75; 88% and 17% of “easy” and “very difficult” ESDs completed &lt;120 minutes, respectively). To predict ESD duration as continuous outcome, we developed and validated the six-variable cESD-TIME formula (https://cesdtimeformula.shinyapps.io/calculator/; optimism-corrected R2=61%; R2=66% after recalibration of the slope). Conclusions We provided two useful tools for predicting colorectal ESD duration at Western centers. Further improvements and validations are encouraged with potential local adaptation to optimize time planning.

Diagnostic management of acute pulmonary embolism: a prediction model based on a patient data meta-analysis.

Risk stratification is used for decisions regarding need for imaging in patients with clinically suspected acute pulmonary embolism (PE). The aim was to develop a clinical prediction model that provides an individualized, accurate probability estimate for the presence of acute PE in patients with suspected disease based on readily available clinical items and D-dimer concentrations. An individual patient data meta-analysis was performed based on sixteen cross-sectional or prospective studies with data from 28 305 adult patients with clinically suspected PE from various clinical settings, including primary care, emergency care, hospitalized and nursing home patients. A multilevel logistic regression model was built and validated including ten a priori defined objective candidate predictors to predict objectively confirmed PE at baseline or venous thromboembolism (VTE) during follow-up of 30 to 90 days. Multiple imputation was used for missing data. Backward elimination was performed with a P-value <0.10. Discrimination (c-statistic with 95% confidence intervals [CI] and prediction intervals [PI]) and calibration (outcome:expected [O:E] ratio and calibration plot) were evaluated based on internal-external cross-validation. The accuracy of the model was subsequently compared with algorithms based on the Wells score and D-dimer testing. The final model included age (in years), sex, previous VTE, recent surgery or immobilization, haemoptysis, cancer, clinical signs of deep vein thrombosis, inpatient status, D-dimer (in µg/L), and an interaction term between age and D-dimer. The pooled c-statistic was 0.87 (95% CI, 0.85-0.89; 95% PI, 0.77-0.93) and overall calibration was very good (pooled O:E ratio, 0.99; 95% CI, 0.87-1.14; 95% PI, 0.55-1.79). The model slightly overestimated VTE probability in the lower range of estimated probabilities. Discrimination of the current model in the validation data sets was better than that of the Wells score combined with a D-dimer threshold based on age (c-statistic 0.73; 95% CI, 0.70-0.75) or structured clinical pretest probability (c-statistic 0.79; 95% CI, 0.76-0.81). The present model provides an absolute, individualized probability of PE presence in a broad population of patients with suspected PE, with very good discrimination and calibration. Its clinical utility needs to be evaluated in a prospective management or impact study. PROSPERO ID 89366.

Postpartum readmission for hypertension and pre-eclampsia: development and validation of a predictive model.

To develop a model for predicting postpartum readmission for hypertension and pre-eclampsia at delivery discharge and assess external validation or model transportability across clinical sites. Prediction model using data available in the electronic health record from two clinical sites. Two tertiary care health systems from the Southern (2014-2015) and Northeastern USA (2017-2019). A total of 28 201 postpartum individuals: 10 100 in the South and 18 101 in the Northeast. An internal-external cross validation (IECV) approach was used to assess external validation or model transportability across the two sites. In IECV, data from each health system were first used to develop and internally validate a prediction model; each model was then externally validated using the other health system. Models were fit using penalised logistic regression, and accuracy was estimated using discrimination (concordance index), calibration curves and decision curves. Internal validation was performed using bootstrapping with bias-corrected performance measures. Decision curve analysis was used to display potential cut points where the model provided net benefit for clinical decision-making. The outcome was postpartum readmission for either hypertension or pre-eclampsia <6 weeks after delivery. The postpartum readmission rate for hypertension and pre-eclampsia overall was 0.9% (0.3% and 1.2% by site, respectively). The final model included six variables: age, parity, maximum postpartum diastolic blood pressure, birthweight, pre-eclampsia before discharge and delivery mode (and interaction between pre-eclampsia × delivery mode). Discrimination was adequate at both health systems on internal validation (c-statistic South: 0.88; 95% confidence interval [CI] 0.87-0.89; Northeast: 0.74; 95% CI 0.74-0.74). In IECV, discrimination was inconsistent across sites, with improved discrimination for the Northeastern model on the Southern cohort (c-statistic 0.61 and 0.86, respectively), but calibration was not adequate. Next, model updating was performed using the combined dataset to develop a new model. This final model had adequate discrimination (c-statistic: 0.80, 95% CI 0.80-0.80), moderate calibration (intercept -0.153, slope 0.960, Emax 0.042) and provided superior net benefit at clinical decision-making thresholds between 1% and 7% for interventions preventing readmission. An online calculator is provided here. Postpartum readmission for hypertension and pre-eclampsia may be accurately predicted but further model validation is needed. Model updating using data from multiple sites will be needed before use across clinical settings.

An Update to the Kaiser Permanente Inpatient Risk Adjustment Methodology Accurately Predicts In-Hospital Mortality: a Retrospective Cohort Study.

Methods to accurately predict the risk of in-hospital mortality are important for applications including quality assessment of healthcare institutions and research. To update and validate the Kaiser Permanente inpatient risk adjustment methodology (KP method) to predict in-hospital mortality, using open-source tools to measure comorbidity and diagnosis groups, and removing troponin which is difficult to standardize across modern clinical assays. Retrospective cohort study using electronic health record data from GEMINI. GEMINI is a research collaborative that collects administrative and clinical data from hospital information systems. Adult general medicine inpatients at 28 hospitals in Ontario, Canada, between April 2010 and December 2022. The outcome was in-hospital mortality, modeled by diagnosis groupusing 56 logistic regressions. We compared models with and without troponin as an input to the laboratory-based acute physiology score. We fit and validated the updated method using internal-external cross-validation at 28 hospitals from April 2015 to December 2022. In 938,103 hospitalizations with 7.2% in-hospital mortality, the updated KP method accurately predicted the risk of mortality. The c-statistic at the median hospital was 0.866 (see Fig. 3) (25th-75th 0.848-0.876, range 0.816-0.927) and calibration was strong for nearly all patients at all hospitals. The 95th percentile absolute difference between predicted and observed probabilities was 0.038 at the median hospital (25th-75th 0.024-0.057, range 0.006-0.118). Model performance was very similar with and without troponin in a subset of 7 hospitals, and performance was similar with and without troponin for patients hospitalized for heart failure and acute myocardial infarction. An update to the KP method accurately predicted in-hospital mortality for general medicine inpatients in 28 hospitals in Ontario, Canada. This updated method can be implemented in a wider range of settings using common open-source tools.

Development and internal-external validation of statistical and machine learning models for breast cancer prognostication: cohort study

ObjectiveTo develop a clinically useful model that estimates the 10 year risk of breast cancer related mortality in women (self-reported female sex) with breast cancer of any stage, comparing results...

Development of treatment-decision algorithms for children evaluated for pulmonary tuberculosis: an individual participant data meta-analysis.

Many children with pulmonary tuberculosis remain undiagnosed and untreated with related high morbidity and mortality. Recent advances in childhood tuberculosis algorithm development have incorporated prediction modelling, but studies so far have been small and localised, with limited generalisability. We aimed to evaluate the performance of currently used diagnostic algorithms and to use prediction modelling to develop evidence-based algorithms to assist in tuberculosis treatment decision making for children presenting to primary health-care centres. For this meta-analysis, we identified individual participant data from a WHO public call for data on the management of tuberculosis in children and adolescents and referral from childhood tuberculosis experts. We included studies that prospectively recruited consecutive participants younger than 10 years attending health-care centres in countries with a high tuberculosis incidence for clinical evaluation of pulmonary tuberculosis. We collated individual participant data including clinical, bacteriological, and radiological information and a standardised reference classification of pulmonary tuberculosis. Using this dataset, we first retrospectively evaluated the performance of several existing treatment-decision algorithms. We then used the data to develop two multivariable prediction models that included features used in clinical evaluation of pulmonary tuberculosis-one with chest x-ray features and one without-and we investigated each model's generalisability using internal-external cross-validation. The parameter coefficient estimates of the two models were scaled into two scoring systems to classify tuberculosis with a prespecified sensitivity target. The two scoring systems were used to develop two pragmatic, treatment-decision algorithms for use in primary health-care settings. Of 4718 children from 13 studies from 12 countries, 1811 (38·4%) were classified as having pulmonary tuberculosis: 541 (29·9%) bacteriologically confirmed and 1270 (70·1%) unconfirmed. Existing treatment-decision algorithms had highly variable diagnostic performance. The scoring system derived from the prediction model that included clinical features and features from chest x-ray had a combined sensitivity of 0·86 [95% CI 0·68-0·94] and specificity of 0·37 [0·15-0·66] against a composite reference standard. The scoring system derived from the model that included only clinical features had a combined sensitivity of 0·84 [95% CI 0·66-0·93] and specificity of 0·30 [0·13-0·56] against a composite reference standard. The scoring system from each model was placed after triage steps, including assessment of illness acuity and risk of poor tuberculosis-related outcomes, to develop treatment-decision algorithms. We adopted an evidence-based approach to develop pragmatic algorithms to guide tuberculosis treatment decisions in children, irrespective of the resources locally available. This approach will empower health workers in primary health-care settings with high tuberculosis incidence and limited resources to initiate tuberculosis treatment in children to improve access to care and reduce tuberculosis-related mortality. These algorithms have been included in the operational handbook accompanying the latest WHO guidelines on the management of tuberculosis in children and adolescents. Future prospective evaluation of algorithms, including those developed in this work, is necessary to investigate clinical performance. WHO, US National Institutes of Health.