Internal-external Cross-validation Research Articles

An Electronic Medical Record-Based Prognostic Model for Inpatient Falls: Development and Internal-External Cross-Validation.

Effective fall prevention interventions in hospitals require appropriate allocation of resources early in admission. To address this, fall risk prediction tools and models have been developed with the aim to provide fall prevention strategies to patients at high risk. However, fall risk assessment tools have typically been inaccurate for prediction, ineffective in prevention, and time-consuming to complete. Accurate, dynamic, individualized estimates of fall risk for admitted patients using routinely recorded data may assist in prioritizing fall prevention efforts. The objective of this study was to develop and validate an accurate and dynamic prognostic model for inpatient falls among a cohort of patients using routinely recorded electronic medical record data. We used routinely recorded data from 5 Australian hospitals to develop and internally-externally validate a prediction model for inpatient falls using a Cox proportional hazards model with time-varying covariates. The study cohort included patients admitted during 2018-2021 to any ward, with no age restriction. Predictors used in the model included admission-related administrative data, length of stay, and number of previous falls during the admission (updated every 12 hours up to 14 days after admission). Model calibration was assessed using Poisson regression and discrimination using the area under the time-dependent receiver operating characteristic curve. There were 1,107,556 inpatient admissions, 6004 falls, and 5341 unique fallers. The area under the time-dependent receiver operating characteristic curve was 0.899 (95% CI 0.88-0.91) at 24 hours after admission and declined throughout admission (eg, 0.765, 95% CI 0.75-0.78 on the seventh day after admission). Site-dependent overestimation and underestimation of risk was observed on the calibration plots. Using a large dataset from multiple hospitals and robust methods to model development and validation, we developed a prognostic model for inpatient falls. It had high discrimination, suggesting the model has the potential for operationalization in clinical decision support for prioritizing inpatients for fall prevention. Performance was site dependent, and model recalibration may lead to improved performance.

Development of a prediction model for clinically-relevant fatigue: a multi-cancer approach.

Fatigue is the most prevalent symptom across cancer types. To support clinicians in providing fatigue-related supportive care, this study aims to develop and compare models predicting clinically relevant fatigue (CRF) occurring between two and three years after diagnosis, and to assess the validity of the best-performing model across diverse cancer populations. Patients with non-metastatic bladder, colorectal, endometrial, ovarian, or prostate cancer who completed a questionnaire within three months after diagnosis and a subsequent questionnaire between two and three years thereafter, were included. Predictor variables included clinical, socio-demographic, and patient-reported variables. The outcome was CRF (EORTC QLQC30 fatigue ≥ 39). Logistic regression using LASSO selection was compared to more advanced Machine Learning (ML) based models, including Extreme gradient boosting (XGBoost), support vector machines (SVM), and artificial neural networks (ANN). Internal-external cross-validation was conducted on the best-performing model. 3160 patients were included. The logistic regression model had the highest C-statistic (0.77) and balanced accuracy (0.65), both indicating good discrimination between patients with and without CRF. However, sensitivity was low across all models (0.22-0.37). Following internal-external validation, performance across cancer types was consistent (C-statistics 0.73-0.82). Although the models' discrimination was good, the low balanced accuracy and poor calibration in the presence of CRF indicates a relatively high likelihood of underdiagnosis of future CRF. Yet, the clinical applicability of the model remains uncertain. The logistic regression performed better than the ML-based models and was robust across cohorts, suggesting an advantage of simpler models to predict CRF.

Predictive Models for Sustained, Uncontrolled Hypertension and Hypertensive Crisis Based on Electronic Health Record Data: Algorithm Development and Validation.

Assessing disease progression among patients with uncontrolled hypertension is important for identifying opportunities for intervention. We aim to develop and validate 2 models, one to predict sustained, uncontrolled hypertension (≥2 blood pressure [BP] readings ≥140/90 mm Hg or ≥1 BP reading ≥180/120 mm Hg) and one to predict hypertensive crisis (≥1 BP reading ≥180/120 mm Hg) within 1 year of an index visit (outpatient or ambulatory encounter in which an uncontrolled BP reading was recorded). Data from 142,897 patients with uncontrolled hypertension within Atrium Health Greater Charlotte in 2018 were used. Electronic health record-based predictors were based on the 1-year period before a patient's index visit. The dataset was randomly split (80:20) into a training set and a validation set. In total, 4 machine learning frameworks were considered: L2-regularized logistic regression, multilayer perceptron, gradient boosting machines, and random forest. Model selection was performed with 10-fold cross-validation. The final models were assessed on discrimination (C-statistic), calibration (eg, integrated calibration index), and net benefit (with decision curve analysis). Additionally, internal-external cross-validation was performed at the county level to assess performance with new populations and summarized using random-effect meta-analyses. In internal validation, the C-statistic and integrated calibration index were 0.72 (95% CI 0.71-0.72) and 0.015 (95% CI 0.012-0.020) for the sustained, uncontrolled hypertension model, and 0.81 (95% CI 0.79-0.82) and 0.009 (95% CI 0.007-0.011) for the hypertensive crisis model. The models had higher net benefit than the default policies (ie, treat-all and treat-none) across different decision thresholds. In internal-external cross-validation, the pooled performance was consistent with internal validation results; in particular, the pooled C-statistics were 0.70 (95% CI 0.69-0.71) and 0.79 (95% CI 0.78-0.81) for the sustained, uncontrolled hypertension model and hypertensive crisis model, respectively. An electronic health record-based model predicted hypertensive crisis reasonably well in internal and internal-external validations. The model can potentially be used to support population health surveillance and hypertension management. Further studies are needed to improve the ability to predict sustained, uncontrolled hypertension.

A prediction model for electroconvulsive therapy effectiveness in patients with major depressive disorder from the Dutch ECT Consortium (DEC).

Reliable predictors for electroconvulsive therapy (ECT) effectiveness would allow a more precise and personalized approach for the treatment of major depressive disorder (MDD). Prediction models were created using a priori selected clinical variables based on previous meta-analyses. Multivariable linear regression analysis was used, applying backwards selection to determine predictor variables while allowing non-linear relations, to develop a prediction model for depression outcome post-ECT (and logistic regression for remission and response as secondary outcome measures). Internal validation and internal-external cross-validation were used to examine overfitting and generalizability of the model's predictive performance. In total, 1892 adult patients with MDD were included from 22 clinical and research cohorts of the twelve sites within the Dutch ECT Consortium. The final primary prediction model showed several factors that significantly predicted a lower depression score post-ECT: higher age, shorter duration of the current depressive episode, severe MDD with psychotic features, lower level of previous antidepressant resistance in the current episode, higher pre-ECT global cognitive functioning, absence of a comorbid personality disorder, and a lower level of failed psychotherapy in the current episode. The optimism-adjusted R² of the final model was 19%. This prediction model based on readily available clinical information can reduce uncertainty of ECT outcomes and hereby inform clinical decision-making, as prompt referral for ECT may be particularly beneficial for individuals with the above-mentioned characteristics. However, despite including a large number of pretreatment factors, a large proportion of the variance in depression outcome post-ECT remained unpredictable.

Risk of Recurrent Venous Thromboembolism in Patients with Cancer: An Individual Patient Data Meta-analysis and Development of a Prediction Model.

About 7% of patients with cancer-associated venous thromboembolism (CAT) develop a recurrence during anticoagulant treatment. Identification of high-risk patients may help guide treatment decisions. To identify clinical predictors and develop a prediction model for on-treatment recurrent CAT. For this individual patient data meta-analysis, we used data from four randomized controlled trials evaluating low-molecular-weight heparin or direct oral anticoagulants (DOACs) for CAT (Hokusai VTE Cancer, SELECT-D, CLOT, and CATCH). The primary outcome was adjudicated on-treatment recurrent CAT during a 6-month follow-up. A clinical prediction model was developed using multivariable logistic regression analysis with backward selection. This model was validated using internal-external cross-validation. Performance was assessed by the c-statistic and a calibration plot. After excluding patients using vitamin K antagonists, the combined dataset comprised 2,245 patients with cancer and acute CAT who were treated with edoxaban (23%), rivaroxaban (9%), dalteparin (47%), or tinzaparin (20%). Recurrent on-treatment CAT during the 6-month follow-up occurred in 150 (6.7%) patients. Predictors included in the final model were age (restricted cubic spline), breast cancer (odds ratio [OR]: 0.42; 95% confidence interval [CI]: 0.20-0.87), metastatic disease (OR: 1.44; 95% CI: 1.01-2.05), treatment with DOAC (OR: 0.66; 95% CI: 0.44-0.98), and deep vein thrombosis only as an index event (OR: 1.72; 95% CI: 1.31-2.27). The c-statistic of the model was 0.63 (95% CI: 0.54-0.72) after internal-external cross-validation. Calibration varied across studies. The prediction model for recurrent CAT included five clinical predictors and has only modest discrimination. Prediction of recurrent CAT at the initiation of anticoagulation remains challenging.

Effect of disability, homelessness, and neighborhood marginalization on risk-adjustment for hospital performance measurement.

Background It is not known how disability, homelessness, or neighborhood marginalization influence risk-adjusted hospital performance measurement in a universal health care system. Methods We evaluated the effect of including these equity-related factors in risk-adjustment models for in-hospital mortality, and 7- and 30-day readmission in 28 hospitals in Ontario, Canada. We compared risk-adjustment with commonly-used clinical factors to models that also included homelessness, disability, and neighborhood indices of marginalization. We evaluated models in historical data using internal-external cross-validation. We calculated risk-standardized outcome rates for each hospital in a recent reporting period using mixed-effects logistic regression. Results The cohort included 544,805 admissions. Adjustment for disability, homelessness, and neighborhood marginalization had little impact on discrimination or calibration of risk-adjustment models. However, it influenced comparative hospital performance on risk-standardized 30-day readmission rates, resulting in 5 hospitals being reclassified between below-average, average, and above-average groups. No hospitals were reclassified for mortality and 7-day readmission. Conclusion In a system with universally insured hospital services, adjustment for disability, homelessness, and neighborhood marginalization influenced estimates of hospital performance for 30-day readmission but not 7-day readmission or in-hospital mortality. These findings can inform researchers and policymakers as they thoughtfully consider when to adjust for these factors in hospital performance measurement.

Sudden cardiac death after myocardial infarction: individual participant data from pooled cohorts.

Risk stratification of sudden cardiac death after myocardial infarction and prevention by defibrillator rely on left ventricular ejection fraction (LVEF). Improved risk stratification across the whole LVEF range is required for decision-making on defibrillator implantation. The analysis pooled 20 data sets with 140 204 post-myocardial infarction patients containing information on demographics, medical history, clinical characteristics, biomarkers, electrocardiography, echocardiography, and cardiac magnetic resonance imaging. Separate analyses were performed in patients (i) carrying a primary prevention cardioverter-defibrillator with LVEF ≤ 35% [implantable cardioverter-defibrillator (ICD) patients], (ii) without cardioverter-defibrillator with LVEF ≤ 35% (non-ICD patients ≤ 35%), and (iii) without cardioverter-defibrillator with LVEF > 35% (non-ICD patients >35%). Primary outcome was sudden cardiac death or, in defibrillator carriers, appropriate defibrillator therapy. Using a competing risk framework and systematic internal-external cross-validation, a model using LVEF only, a multivariable flexible parametric survival model, and a multivariable random forest survival model were developed and externally validated. Predictive performance was assessed by random effect meta-analysis. There were 1326 primary outcomes in 7543 ICD patients, 1193 in 25 058 non-ICD patients ≤35%, and 1567 in 107 603 non-ICD patients >35% during mean follow-up of 30.0, 46.5, and 57.6 months, respectively. In these three subgroups, LVEF poorly predicted sudden cardiac death (c-statistics between 0.50 and 0.56). Considering additional parameters did not improve calibration and discrimination, and model generalizability was poor. More accurate risk stratification for sudden cardiac death and identification of low-risk individuals with severely reduced LVEF or of high-risk individuals with preserved LVEF was not feasible, neither using LVEF nor using other predictors.

Development and validation of a prognostic model to predict relapse in adults with remitted depression in primary care: secondary analysis of pooled individual participant data from multiple studies.

Relapse of depression is common and contributes to the overall associated morbidity and burden. We lack evidence-based tools to estimate an individual's risk of relapse after treatment in primary care, which may help us more effectively target relapse prevention. The objective was to develop and validate a prognostic model to predict risk of relapse of depression in primary care. Multilevel logistic regression models were developed, using individual participant data from seven primary care-based studies (n=1244), to predict relapse of depression. The model was internally validated using bootstrapping, and generalisability was explored using internal-external cross-validation. Residual depressive symptoms (OR: 1.13 (95% CI: 1.07 to 1.20), p<0.001) and baseline depression severity (OR: 1.07 (1.04 to 1.11), p<0.001) were associated with relapse. The validated model had low discrimination (C-statistic 0.60 (0.55-0.65)) and miscalibration concerns (calibration slope 0.81 (0.31-1.31)). On secondary analysis, being in a relationship was associated with reduced risk of relapse (OR: 0.43 (0.28-0.67), p<0.001); this remained statistically significant after correction for multiple significance testing. We could not predict risk of depression relapse with sufficient accuracy in primary care data, using routinely recorded measures. Relationship status warrants further research to explore its role as a prognostic factor for relapse. Until we can accurately stratify patients according to risk of relapse, a universal approach to relapse prevention may be most beneficial, either during acute-phase treatment or post remission. Where possible, this could be guided by the presence or absence of known prognostic factors (eg, residual depressive symptoms) and targeted towards these. NCT04666662.

Diagnostic Prediction Model for Tuberculous Meningitis: An Individual Participant Data Meta-Analysis.

No accurate and rapid diagnostic test exists for tuberculous meningitis (TBM), leading to delayed diagnosis. We leveraged data from multiple studies to improve the predictive performance of diagnostic models across different populations, settings, and subgroups to develop a new predictive tool for TBM diagnosis. We conducted a systematic review to analyze eligible datasets with individual-level participant data (IPD). We imputed missing data and explored three approaches: stepwise logistic regression, classification and regression tree (CART), and random forest regression. We evaluated performance using calibration plots and C-statistics via internal-external cross-validation. We included 3,761 individual participants from 14 studies and nine countries. A total of 1,240 (33%) participants had "definite" (30%) or "probable" (3%) TBM by case definition. Important predictive variables included cerebrospinal fluid (CSF) glucose, blood glucose, CSF white cell count, CSF differential, cryptococcal antigen, HIV status, and fever presence. Internal validation showed that performance varied considerably between IPD datasets with C-statistic values between 0.60 and 0.89. In external validation, CART performed the worst (C = 0.82), and logistic regression and random forest had the same accuracy (C = 0.91). We developed a mobile app for TBM clinical prediction that accounted for heterogeneity and improved diagnostic performance (https://tbmcalc.github.io/tbmcalc). Further external validation is needed.

Development and validation of a prognostic model to predict birth weight: individual participant data meta-analysis

ObjectiveTo predict birth weight at various potential gestational ages of delivery based on data routinely available at the first antenatal visit.DesignIndividual participant data meta-analysis.Data sourcesIndividual participant data of four cohorts...

Development and validation of prediction models for fetal growth restriction and birthweight: an individual participant data meta-analysis.

Fetal growth restriction is associated with perinatal morbidity and mortality. Early identification of women having at-risk fetuses can reduce perinatal adverse outcomes. To assess the predictive performance of existing models predicting fetal growth restriction and birthweight, and if needed, to develop and validate new multivariable models using individual participant data. Individual participant data meta-analyses of cohorts in International Prediction of Pregnancy Complications network, decision curve analysis and health economics analysis. Pregnant women at booking. External validation of existing models (9 cohorts, 441,415 pregnancies); International Prediction of Pregnancy Complications model development and validation (4 cohorts, 237,228 pregnancies). Maternal clinical characteristics, biochemical and ultrasound markers. fetal growth restriction defined as birthweight <10th centile adjusted for gestational age and with stillbirth, neonatal death or delivery before 32 weeks' gestation birthweight. First, we externally validated existing models using individual participant data meta-analysis. If needed, we developed and validated new International Prediction of Pregnancy Complications models using random-intercept regression models with backward elimination for variable selection and undertook internal-external cross-validation. We estimated the study-specific performance (c-statistic, calibration slope, calibration-in-the-large) for each model and pooled using random-effects meta-analysis. Heterogeneity was quantified using τ2 and 95% prediction intervals. We assessed the clinical utility of the fetal growth restriction model using decision curve analysis, and health economics analysis based on National Institute for Health and Care Excellence 2008 model. Of the 119 published models, one birthweight model (Poon) could be validated. None reported fetal growth restriction using our definition. Across all cohorts, the Poon model had good summary calibration slope of 0.93 (95% confidence interval 0.90 to 0.96) with slight overfitting, and underpredicted birthweight by 90.4 g on average (95% confidence interval 37.9 g to 142.9 g). The newly developed International Prediction of Pregnancy Complications-fetal growth restriction model included maternal age, height, parity, smoking status, ethnicity, and any history of hypertension, pre-eclampsia, previous stillbirth or small for gestational age baby and gestational age at delivery. This allowed predictions conditional on a range of assumed gestational ages at delivery. The pooled apparent c-statistic and calibration were 0.96 (95% confidence interval 0.51 to 1.0), and 0.95 (95% confidence interval 0.67 to 1.23), respectively. The model showed positive net benefit for predicted probability thresholds between 1% and 90%. In addition to the predictors in the International Prediction of Pregnancy Complications-fetal growth restriction model, the International Prediction of Pregnancy Complications-birthweight model included maternal weight, history of diabetes and mode of conception. Average calibration slope across cohorts in the internal-external cross-validation was 1.00 (95% confidence interval 0.78 to 1.23) with no evidence of overfitting. Birthweight was underestimated by 9.7 g on average (95% confidence interval -154.3 g to 173.8 g). We could not externally validate most of the published models due to variations in the definitions of outcomes. Internal-external cross-validation of our International Prediction of Pregnancy Complications-fetal growth restriction model was limited by the paucity of events in the included cohorts. The economic evaluation using the published National Institute for Health and Care Excellence 2008 model may not reflect current practice, and full economic evaluation was not possible due to paucity of data. International Prediction of Pregnancy Complications models' performance needs to be assessed in routine practice, and their impact on decision-making and clinical outcomes needs evaluation. The International Prediction of Pregnancy Complications-fetal growth restriction and International Prediction of Pregnancy Complications-birthweight models accurately predict fetal growth restriction and birthweight for various assumed gestational ages at delivery. These can be used to stratify the risk status at booking, plan monitoring and management. This study is registered as PROSPERO CRD42019135045. This award was funded by the National Institute for Health and Care Research (NIHR) Health Technology Assessment programme (NIHR award ref: 17/148/07) and is published in full in Health Technology Assessment; Vol. 28, No. 14. See the NIHR Funding and Awards website for further award information.

Risk of serious adverse events after primary shoulder replacement: development and external validation of a prediction model using linked national data from England and Denmark

Despite a rising rate of serious medical complications after shoulder replacement surgery, there are no prediction models in widespread use to guide surgeons in identifying patients at high risk and to provide patients with personalised risk estimates to support shared decision making. Our aim was to develop and externally validate a prediction model for serious adverse events within 90 days of primary shoulder replacement surgery. Linked data from the National Joint Registry, National Health Service Hospital Episode Statistics Admitted Patient Care of England, and Civil Registration Mortality databases and Danish Shoulder Arthroplasty Registry and National Patient Register were used for our modelling study. Patients aged 18-100 years who had a primary shoulder replacement between April 1, 2012, and Oct 2, 2020, in England, and April 1, 2012, and Oct 2, 2018, in Denmark, were included. We developed a multivariable logistic regression model using the English dataset to predict the risk of 90-day serious adverse events, which were defined as medical complications requiring admission to hospital and all-cause death. We undertook internal validation using bootstrapping, and internal-external cross-validation across different geographical regions of England. The English model was externally validated on the Danish dataset. Data for 40 631 patients undergoing primary shoulder replacement (mean age 72·5 years [SD 9·9]; 28 709 [70·7%] women and 11 922 [29·3%] men) were used for model development, of whom 2270 (5·6%) had a 90-day serious adverse event. On internal validation, the model had a C-statistic of 0·717 (95% CI 0·707-0·728) and was well calibrated. Internal-external cross-validation showed consistent model performance across all regions in England. Upon external validation on the Danish dataset (n=6653; mean age 70·5 years [SD 10·3]; 4503 [67·7%] women and 2150 [32·3%] men), the model had a C-statistic of 0·750 (95% CI 0·723-0·776). Decision curve analysis showed clinical utility, with net benefit across all risk thresholds. This externally validated prediction model uses commonly available clinical variables to accurately predict the risk of serious medical complications after primary shoulder replacement surgery. The model is generalisable and applicable to most patients in need of a shoulder replacement. Its use offers support to clinicians and could inform and empower patients in the shared decision-making process. National Institute for Health and Care Research and the Department of Orthopaedic Surgery, Herlev and Gentofte Hospital, Denmark.

Prioritising deteriorating patients using time-to-event analysis: prediction model development and internal–external validation

BackgroundBinary classification models are frequently used to predict clinical deterioration, however they ignore information on the timing of events. An alternative is to apply time-to-event models, augmenting clinical workflows by ranking patients by predicted risks. This study examines how and why time-to-event modelling of vital signs data can help prioritise deterioration assessments using lift curves, and develops a prediction model to stratify acute care inpatients by risk of clinical deterioration.MethodsWe developed and validated a Cox regression for time to in-hospital mortality. The model used time-varying covariates to estimate the risk of clinical deterioration. Adult inpatient medical records from 5 Australian hospitals between 1 January 2019 and 31 December 2020 were used for model development and validation. Model discrimination and calibration were assessed using internal–external cross validation. A discrete-time logistic regression model predicting death within 24 h with the same covariates was used as a comparator to the Cox regression model to estimate differences in predictive performance between the binary and time-to-event outcome modelling approaches.ResultsOur data contained 150,342 admissions and 1016 deaths. Model discrimination was higher for Cox regression than for discrete-time logistic regression, with cross-validated AUCs of 0.96 and 0.93, respectively, for mortality predictions within 24 h, declining to 0.93 and 0.88, respectively, for mortality predictions within 1 week. Calibration plots showed that calibration varied by hospital, but this can be mitigated by ranking patients by predicted risks.ConclusionTime-varying covariate Cox models can be powerful tools for triaging patients, which may lead to more efficient and effective care in time-poor environments when the times between observations are highly variable.

Musculoskeletal Health and Work: Development and Internal-External Cross-Validation of a Model to Predict Risk of Work Absence and Presenteeism in People Seeking Primary Healthcare.

To develop and validate prediction models for the risk of future work absence and level of presenteeism, in adults seeking primary healthcare with musculoskeletal disorders (MSD). Six studies from the West-Midlands/Northwest regions of England, recruiting adults consulting primary care with MSD were included for model development and internal-external cross-validation (IECV). The primary outcome was any work absence within 6 months of their consultation. Secondary outcomes included 6-month presenteeism and 12-month work absence. Ten candidate predictors were included: age; sex; multisite pain; baseline pain score; pain duration; job type; anxiety/depression; comorbidities; absence in the previous 6 months; and baseline presenteeism. For the 6-month absence model, 2179 participants (215 absences) were available across five studies. Calibration was promising, although varied across studies, with a pooled calibration slope of 0.93 (95% CI: 0.41-1.46) on IECV. On average, the model discriminated well between those with work absence within 6 months, and those without (IECV-pooled C-statistic 0.76, 95% CI: 0.66-0.86). The 6-month presenteeism model, while well calibrated on average, showed some individual-level variation in predictive accuracy, and the 12-month absence model was poorly calibrated due to the small available size for model development. The developed models predict 6-month work absence and presenteeism with reasonable accuracy, on average, in adults consulting with MSD. The model to predict 12-month absence was poorly calibrated and is not yet ready for use in practice. This information may support shared decision-making and targeting occupational health interventions at those with a higher risk of absence or presenteeism in the 6 months following consultation. Further external validation is needed before the models' use can be recommended or their impact on patients can be fully assessed.

Artificial Intelligence-based Segmentation of Residual Pancreatic Cancer in Resection Specimens Following Neoadjuvant Treatment (ISGPP-2): International Improvement and Validation Study.

Neoadjuvant therapy (NAT) has become routine in patients with borderline resectable pancreatic cancer. Pathologists examine pancreatic cancer resection specimens to evaluate the effect of NAT. However, an automated scoring system to objectively quantify residual pancreatic cancer (RPC) is currently lacking. Herein, we developed and validated the first automated segmentation model using artificial intelligence techniques to objectively quantify RPC. Digitized histopathological tissue slides were included from resected pancreatic cancer specimens from 14 centers in 7 countries in Europe, North America, Australia, and Asia. Four different scanner types were used: Philips (56%), Hamamatsu (27%), 3DHistech (10%), and Leica (7%). Regions of interest were annotated and classified as cancer, non-neoplastic pancreatic ducts, and others. A U-Net model was trained to detect RPC. Validation consisted of by-scanner internal-external cross-validation. Overall, 528 unique hematoxylin and eosin (H & E) slides from 528 patients were included. In the individual Philips, Hamamatsu, 3DHistech, and Leica scanner cross-validations, mean F1 scores of 0.81 (95% CI, 0.77-0.84), 0.80 (0.78-0.83), 0.76 (0.65-0.78), and 0.71 (0.65-0.78) were achieved, respectively. In the meta-analysis of the cross-validations, the mean F1 score was 0.78 (0.71-0.84). A final model was trained on the entire data set. This ISGPP model is the first segmentation model using artificial intelligence techniques to objectively quantify RPC following NAT. The internally-externally cross-validated model in this study demonstrated robust performance in detecting RPC in specimens. The ISGPP model, now made publically available, enables automated RPC segmentation and forms the basis for objective NAT response evaluation in pancreatic cancer.

A panoptic segmentation dataset and deep-learning approach for explainable scoring of tumor-infiltrating lymphocytes

Tumor-Infiltrating Lymphocytes (TILs) have strong prognostic and predictive value in breast cancer, but their visual assessment is subjective. To improve reproducibility, the International Immuno-oncology Working Group recently released recommendations for the computational assessment of TILs that build on visual scoring guidelines. However, existing resources do not adequately address these recommendations due to the lack of annotation datasets that enable joint, panoptic segmentation of tissue regions and cells. Moreover, existing deep-learning methods focus entirely on either tissue segmentation or cell nuclei detection, which complicates the process of TILs assessment by necessitating the use of multiple models and reconciling inconsistent predictions. We introduce PanopTILs, a region and cell-level annotation dataset containing 814,886 nuclei from 151 patients, openly accessible at: sites.google.com/view/panoptils. Using PanopTILs we developed MuTILs, a neural network optimized for assessing TILs in accordance with clinical recommendations. MuTILs is a concept bottleneck model designed to be interpretable and to encourage sensible predictions at multiple resolutions. Using a rigorous internal-external cross-validation procedure, MuTILs achieves an AUROC of 0.93 for lymphocyte detection and a DICE coefficient of 0.81 for tumor-associated stroma segmentation. Our computational score closely matched visual scores from 2 pathologists (Spearman R = 0.58–0.61, p < 0.001). Moreover, computational TILs scores had a higher prognostic value than visual scores, independent of TNM stage and patient age. In conclusion, we introduce a comprehensive open data resource and a modeling approach for detailed mapping of the breast tumor microenvironment.

Identifying patients with undiagnosed small intestinal neuroendocrine tumours in primary care using statistical and machine learning: model development and validation study

BackgroundNeuroendocrine tumours (NETs) are increasing in incidence, often diagnosed at advanced stages, and individuals may experience years of diagnostic delay, particularly when arising from the small intestine (SI). Clinical prediction models could present novel opportunities for case finding in primary care.MethodsAn open cohort of adults (18+ years) contributing data to the Optimum Patient Care Research Database between 1st Jan 2000 and 30th March 2023 was identified. This database collects de-identified data from general practices in the UK. Model development approaches comprised logistic regression, penalised regression, and XGBoost. Performance (discrimination and calibration) was assessed using internal-external cross-validation. Decision analysis curves compared clinical utility.ResultsOf 11.7 million individuals, 382 had recorded SI NET diagnoses (0.003%). The XGBoost model had the highest AUC (0.869, 95% confidence interval [CI]: 0.841–0.898) but was mildly miscalibrated (slope 1.165, 95% CI: 1.088–1.243; calibration-in-the-large 0.010, 95% CI: −0.164 to 0.185). Clinical utility was similar across all models.DiscussionMultivariable prediction models may have clinical utility in identifying individuals with undiagnosed SI NETs using information in their primary care records. Further evaluation including external validation and health economics modelling may identify cost-effective strategies for case finding for this uncommon tumour.

Development and Validation of a Prediction Model for 1-Year Mortality in Patients With a Hematologic Malignancy Admitted to the ICU.

To develop and validate a prediction model for 1-year mortality in patients with a hematologic malignancy acutely admitted to the ICU. A retrospective cohort study. Five university hospitals in the Netherlands between 2002 and 2015. A total of 1097 consecutive patients with a hematologic malignancy were acutely admitted to the ICU for at least 24 h. None. We created a 13-variable model from 22 potential predictors. Key predictors included active disease, age, previous hematopoietic stem cell transplantation, mechanical ventilation, lowest platelet count, acute kidney injury, maximum heart rate, and type of malignancy. A bootstrap procedure reduced overfitting and improved the model's generalizability. This involved estimating the optimism in the initial model and shrinking the regression coefficients accordingly in the final model. We assessed performance using internal-external cross-validation by center and compared it with the Acute Physiology and Chronic Health Evaluation II model. Additionally, we evaluated clinical usefulness through decision curve analysis. The overall 1-year mortality rate observed in the study was 62% (95% CI, 59-65). Our 13-variable prediction model demonstrated acceptable calibration and discrimination at internal-external validation across centers (C-statistic 0.70; 95% CI, 0.63-0.77), outperforming the Acute Physiology and Chronic Health Evaluation II model (C-statistic 0.61; 95% CI, 0.57-0.65). Decision curve analysis indicated overall net benefit within a clinically relevant threshold probability range of 60-100% predicted 1-year mortality. Our newly developed 13-variable prediction model predicts 1-year mortality in hematologic malignancy patients admitted to the ICU more accurately than the Acute Physiology and Chronic Health Evaluation II model. This model may aid in shared decision-making regarding the continuation of ICU care and end-of-life considerations.

#2801 Which patients may benefit most from hemodiafiltration compared with hemodialysis: prediction of the individualized treatment effect

Abstract Background and Aims Haemodiafiltration (HDF) with an appropriate dosing reduces death from any cause based on an individual participant data meta-analysis and aggregated data meta-analyses of available randomized trials. No differences in treatment effect are reported for specific patient populations when defined on a single characteristic in subgroup analyses. Yet, it is unclear to what extent individual patients, characterized by a combination of clinical characteristics, may benefit from HDF. This study aimed to develop and internally validate a treatment effect prediction model to determine which individual patients would benefit most from well dosed HDF, compared with haemodialysis (HD), in terms of survival. Method Individual participant data from five randomized controlled trials (CONTRAST, ESHOL, Turkish HDF study, French HDF study, CONVINCE) comparing HDF with HD on all-cause mortality were used to derive a Royston-Parmar model for exploring the prediction of absolute treatment effect of HDF based on pre-specified patient and disease characteristics, notably age, sex, body mass index, diabetes mellitus, history of cardiovascular disease, creatinine levels, and c-reactive protein levels. Internal validation of the model was performed using internal-external cross validation. Results Among 4153 participants, with a median follow-up of 30 months (Q1-Q3: 24-36), death from any cause occurred in 558 patients (27.2%). The median predicted survival benefit of HDF compared with HD was 6.9 (Q1–Q3: 5.6–9.0) months, with a range of 2 to 42 months. Patients who were predicted to benefit most from HDF were younger, less likely to have diabetes or a cardiovascular history and had higher serum creatinine levels. Internal-external cross validation showed adequate discrimination and calibration. Conclusion All-cause mortality is reduced by HDF compared with HD in ESKD patients. Yet, the absolute survival benefit seems to considerably differ between individual patients. Our results suggest that the effects of HDF on absolute survival can be predicted accurately using a combination of readily available patient and disease characteristics. This prediction model approach exemplifies the potential of prediction algorithms in supporting clinical decision-making for the choice of treatment options. Validation and potential updates to the presented algorithm are necessary, along with a preferable randomized evaluation of its clinical impact before widespread implementation.

Predicting the risk of pancreatic cancer in adults with new-onset diabetes: development and internal–external validation of a clinical risk prediction model

BackgroundThe National Institute for Health and Care Excellence (NICE) recommends that people aged 60+ years with newly diagnosed diabetes and weight loss undergo abdominal imaging to assess for pancreatic cancer. More nuanced stratification could lead to enrichment of these referral pathways.MethodsPopulation-based cohort study of adults aged 30–85 years at type 2 diabetes diagnosis (2010–2021) using the QResearch primary care database in England linked to secondary care data, the national cancer registry and mortality registers. Clinical prediction models were developed to estimate risks of pancreatic cancer diagnosis within 2 years and evaluated using internal–external cross-validation.ResultsSeven hundred and sixty-seven of 253,766 individuals were diagnosed with pancreatic cancer within 2 years. Models included age, sex, BMI, prior venous thromboembolism, digoxin prescription, HbA1c, ALT, creatinine, haemoglobin, platelet count; and the presence of abdominal pain, weight loss, jaundice, heartburn, indigestion or nausea (previous 6 months). The Cox model had the highest discrimination (Harrell’s C-index 0.802 (95% CI: 0.797–0.817)), the highest clinical utility, and was well calibrated. The model’s highest 1% of predicted risks captured 12.51% of pancreatic cancer cases. NICE guidance had 3.95% sensitivity.DiscussionA new prediction model could have clinical utility in identifying individuals with recent onset diabetes suitable for fast-track abdominal imaging.