Internal-external Cross-validation Research Articles

Development and internal validation of a clinical prediction model for serious complications after emergency laparotomy

PurposeEmergency laparotomy (EL) is a common operation with high risk for postoperative complications, thereby requiring accurate risk stratification to manage vulnerable patients optimally. We developed and internally validated a predictive model of serious complications after EL.MethodsData for eleven carefully selected candidate predictors of 30-day postoperative complications (Clavien-Dindo grade > = 3) were extracted from the HELAS cohort of EL patients in 11 centres in Greece and Cyprus. Logistic regression with Least Absolute Shrinkage and Selection Operator (LASSO) was applied for model development. Discrimination and calibration measures were estimated and clinical utility was explored with decision curve analysis (DCA). Reproducibility and heterogeneity were examined with Bootstrap-based internal validation and Internal–External Cross-Validation. The American College of Surgeons National Surgical Quality Improvement Program’s (ACS-NSQIP) model was applied to the same cohort to establish a benchmark for the new model.ResultsFrom data on 633 eligible patients (175 complication events), the SErious complications After Laparotomy (SEAL) model was developed with 6 predictors (preoperative albumin, blood urea nitrogen, American Society of Anaesthesiology score, sepsis or septic shock, dependent functional status, and ascites). SEAL had good discriminative ability (optimism-corrected c-statistic: 0.80, 95% confidence interval [CI] 0.79–0.81), calibration (optimism-corrected calibration slope: 1.01, 95% CI 0.99–1.03) and overall fit (scaled Brier score: 25.1%, 95% CI 24.1–26.1%). SEAL compared favourably with ACS-NSQIP in all metrics, including DCA across multiple risk thresholds.ConclusionSEAL is a simple and promising model for individualized risk predictions of serious complications after EL. Future external validations should appraise SEAL’s transportability across diverse settings.

Predicting procedure duration of colorectal endoscopic submucosal dissection at Western endoscopy centers

Abstract Background and study aims Overcoming logistical obstacles for the implementation of colorectal endoscopic submucosal dissection (ESD) requires accurate prediction of procedure times. We aimed to evaluate existing and new prediction models for ESD duration. Patients and methods Records of all consecutive patients who underwent single, non-hybrid colorectal ESDs before 2020 at three Dutch centers were reviewed. The performance of an Eastern prediction model [GIE 2021;94(1):133–144] was assessed in the Dutch cohort. A prediction model for procedure duration was built using multivariable linear regression. The model’s performance was validated using internal validation by bootstrap resampling, internal-external cross-validation and external validation in an independent Swedish ESD cohort. Results A total of 435 colorectal ESDs were analyzed (92% en bloc resections, mean duration 139 minutes, mean tumor size 39 mm). The performance of current unstandardized time scheduling practice was suboptimal (explained variance: R2=27%). We successfully validated the Eastern prediction model for colorectal ESD duration <60 minutes (c-statistic 0.70, 95% CI 0.62–0.77), but this model was limited due to dichotomization of the outcome and a relatively low frequency (14%) of ESDs completed <60 minutes in the Dutch centers. The model was more useful with a dichotomization cut-off of 120 minutes (c-statistic: 0.75; 88% and 17% of “easy” and “very difficult” ESDs completed <120 minutes, respectively). To predict ESD duration as continuous outcome, we developed and validated the six-variable cESD-TIME formula (https://cesdtimeformula.shinyapps.io/calculator/; optimism-corrected R2=61%; R2=66% after recalibration of the slope). Conclusions We provided two useful tools for predicting colorectal ESD duration at Western centers. Further improvements and validations are encouraged with potential local adaptation to optimize time planning.

Diagnostic management of acute pulmonary embolism: a prediction model based on a patient data meta-analysis.

Risk stratification is used for decisions regarding need for imaging in patients with clinically suspected acute pulmonary embolism (PE). The aim was to develop a clinical prediction model that provides an individualized, accurate probability estimate for the presence of acute PE in patients with suspected disease based on readily available clinical items and D-dimer concentrations. An individual patient data meta-analysis was performed based on sixteen cross-sectional or prospective studies with data from 28 305 adult patients with clinically suspected PE from various clinical settings, including primary care, emergency care, hospitalized and nursing home patients. A multilevel logistic regression model was built and validated including ten a priori defined objective candidate predictors to predict objectively confirmed PE at baseline or venous thromboembolism (VTE) during follow-up of 30 to 90 days. Multiple imputation was used for missing data. Backward elimination was performed with a P-value <0.10. Discrimination (c-statistic with 95% confidence intervals [CI] and prediction intervals [PI]) and calibration (outcome:expected [O:E] ratio and calibration plot) were evaluated based on internal-external cross-validation. The accuracy of the model was subsequently compared with algorithms based on the Wells score and D-dimer testing. The final model included age (in years), sex, previous VTE, recent surgery or immobilization, haemoptysis, cancer, clinical signs of deep vein thrombosis, inpatient status, D-dimer (in µg/L), and an interaction term between age and D-dimer. The pooled c-statistic was 0.87 (95% CI, 0.85-0.89; 95% PI, 0.77-0.93) and overall calibration was very good (pooled O:E ratio, 0.99; 95% CI, 0.87-1.14; 95% PI, 0.55-1.79). The model slightly overestimated VTE probability in the lower range of estimated probabilities. Discrimination of the current model in the validation data sets was better than that of the Wells score combined with a D-dimer threshold based on age (c-statistic 0.73; 95% CI, 0.70-0.75) or structured clinical pretest probability (c-statistic 0.79; 95% CI, 0.76-0.81). The present model provides an absolute, individualized probability of PE presence in a broad population of patients with suspected PE, with very good discrimination and calibration. Its clinical utility needs to be evaluated in a prospective management or impact study. PROSPERO ID 89366.

Postpartum readmission for hypertension and pre-eclampsia: development and validation of a predictive model.

To develop a model for predicting postpartum readmission for hypertension and pre-eclampsia at delivery discharge and assess external validation or model transportability across clinical sites. Prediction model using data available in the electronic health record from two clinical sites. Two tertiary care health systems from the Southern (2014-2015) and Northeastern USA (2017-2019). A total of 28 201 postpartum individuals: 10 100 in the South and 18 101 in the Northeast. An internal-external cross validation (IECV) approach was used to assess external validation or model transportability across the two sites. In IECV, data from each health system were first used to develop and internally validate a prediction model; each model was then externally validated using the other health system. Models were fit using penalised logistic regression, and accuracy was estimated using discrimination (concordance index), calibration curves and decision curves. Internal validation was performed using bootstrapping with bias-corrected performance measures. Decision curve analysis was used to display potential cut points where the model provided net benefit for clinical decision-making. The outcome was postpartum readmission for either hypertension or pre-eclampsia <6 weeks after delivery. The postpartum readmission rate for hypertension and pre-eclampsia overall was 0.9% (0.3% and 1.2% by site, respectively). The final model included six variables: age, parity, maximum postpartum diastolic blood pressure, birthweight, pre-eclampsia before discharge and delivery mode (and interaction between pre-eclampsia × delivery mode). Discrimination was adequate at both health systems on internal validation (c-statistic South: 0.88; 95% confidence interval [CI] 0.87-0.89; Northeast: 0.74; 95% CI 0.74-0.74). In IECV, discrimination was inconsistent across sites, with improved discrimination for the Northeastern model on the Southern cohort (c-statistic 0.61 and 0.86, respectively), but calibration was not adequate. Next, model updating was performed using the combined dataset to develop a new model. This final model had adequate discrimination (c-statistic: 0.80, 95% CI 0.80-0.80), moderate calibration (intercept -0.153, slope 0.960, Emax 0.042) and provided superior net benefit at clinical decision-making thresholds between 1% and 7% for interventions preventing readmission. An online calculator is provided here. Postpartum readmission for hypertension and pre-eclampsia may be accurately predicted but further model validation is needed. Model updating using data from multiple sites will be needed before use across clinical settings.

An Update to the Kaiser Permanente Inpatient Risk Adjustment Methodology Accurately Predicts In-Hospital Mortality: a Retrospective Cohort Study.

Methods to accurately predict the risk of in-hospital mortality are important for applications including quality assessment of healthcare institutions and research. To update and validate the Kaiser Permanente inpatient risk adjustment methodology (KP method) to predict in-hospital mortality, using open-source tools to measure comorbidity and diagnosis groups, and removing troponin which is difficult to standardize across modern clinical assays. Retrospective cohort study using electronic health record data from GEMINI. GEMINI is a research collaborative that collects administrative and clinical data from hospital information systems. Adult general medicine inpatients at 28 hospitals in Ontario, Canada, between April 2010 and December 2022. The outcome was in-hospital mortality, modeled by diagnosis groupusing 56 logistic regressions. We compared models with and without troponin as an input to the laboratory-based acute physiology score. We fit and validated the updated method using internal-external cross-validation at 28 hospitals from April 2015 to December 2022. In 938,103 hospitalizations with 7.2% in-hospital mortality, the updated KP method accurately predicted the risk of mortality. The c-statistic at the median hospital was 0.866 (see Fig. 3) (25th-75th 0.848-0.876, range 0.816-0.927) and calibration was strong for nearly all patients at all hospitals. The 95th percentile absolute difference between predicted and observed probabilities was 0.038 at the median hospital (25th-75th 0.024-0.057, range 0.006-0.118). Model performance was very similar with and without troponin in a subset of 7 hospitals, and performance was similar with and without troponin for patients hospitalized for heart failure and acute myocardial infarction. An update to the KP method accurately predicted in-hospital mortality for general medicine inpatients in 28 hospitals in Ontario, Canada. This updated method can be implemented in a wider range of settings using common open-source tools.

Development and internal-external validation of statistical and machine learning models for breast cancer prognostication: cohort study

ObjectiveTo develop a clinically useful model that estimates the 10 year risk of breast cancer related mortality in women (self-reported female sex) with breast cancer of any stage, comparing results...

Development of treatment-decision algorithms for children evaluated for pulmonary tuberculosis: an individual participant data meta-analysis.

Many children with pulmonary tuberculosis remain undiagnosed and untreated with related high morbidity and mortality. Recent advances in childhood tuberculosis algorithm development have incorporated prediction modelling, but studies so far have been small and localised, with limited generalisability. We aimed to evaluate the performance of currently used diagnostic algorithms and to use prediction modelling to develop evidence-based algorithms to assist in tuberculosis treatment decision making for children presenting to primary health-care centres. For this meta-analysis, we identified individual participant data from a WHO public call for data on the management of tuberculosis in children and adolescents and referral from childhood tuberculosis experts. We included studies that prospectively recruited consecutive participants younger than 10 years attending health-care centres in countries with a high tuberculosis incidence for clinical evaluation of pulmonary tuberculosis. We collated individual participant data including clinical, bacteriological, and radiological information and a standardised reference classification of pulmonary tuberculosis. Using this dataset, we first retrospectively evaluated the performance of several existing treatment-decision algorithms. We then used the data to develop two multivariable prediction models that included features used in clinical evaluation of pulmonary tuberculosis-one with chest x-ray features and one without-and we investigated each model's generalisability using internal-external cross-validation. The parameter coefficient estimates of the two models were scaled into two scoring systems to classify tuberculosis with a prespecified sensitivity target. The two scoring systems were used to develop two pragmatic, treatment-decision algorithms for use in primary health-care settings. Of 4718 children from 13 studies from 12 countries, 1811 (38·4%) were classified as having pulmonary tuberculosis: 541 (29·9%) bacteriologically confirmed and 1270 (70·1%) unconfirmed. Existing treatment-decision algorithms had highly variable diagnostic performance. The scoring system derived from the prediction model that included clinical features and features from chest x-ray had a combined sensitivity of 0·86 [95% CI 0·68-0·94] and specificity of 0·37 [0·15-0·66] against a composite reference standard. The scoring system derived from the model that included only clinical features had a combined sensitivity of 0·84 [95% CI 0·66-0·93] and specificity of 0·30 [0·13-0·56] against a composite reference standard. The scoring system from each model was placed after triage steps, including assessment of illness acuity and risk of poor tuberculosis-related outcomes, to develop treatment-decision algorithms. We adopted an evidence-based approach to develop pragmatic algorithms to guide tuberculosis treatment decisions in children, irrespective of the resources locally available. This approach will empower health workers in primary health-care settings with high tuberculosis incidence and limited resources to initiate tuberculosis treatment in children to improve access to care and reduce tuberculosis-related mortality. These algorithms have been included in the operational handbook accompanying the latest WHO guidelines on the management of tuberculosis in children and adolescents. Future prospective evaluation of algorithms, including those developed in this work, is necessary to investigate clinical performance. WHO, US National Institutes of Health.

Clinical prediction model for prognosis in kidney transplant recipients (KIDMO): study protocol

BackgroundMany potential prognostic factors for predicting kidney transplantation outcomes have been identified. However, in Switzerland, no widely accepted prognostic model or risk score for transplantation outcomes is being routinely used in clinical practice yet. We aim to develop three prediction models for the prognosis of graft survival, quality of life, and graft function following transplantation in Switzerland.MethodsThe clinical kidney prediction models (KIDMO) are developed with data from a national multi-center cohort study (Swiss Transplant Cohort Study; STCS) and the Swiss Organ Allocation System (SOAS). The primary outcome is the kidney graft survival (with death of recipient as competing risk); the secondary outcomes are the quality of life (patient-reported health status) at 12 months and estimated glomerular filtration rate (eGFR) slope. Organ donor, transplantation, and recipient-related clinical information will be used as predictors at the time of organ allocation. We will use a Fine & Gray subdistribution model and linear mixed-effects models for the primary and the two secondary outcomes, respectively. Model optimism, calibration, discrimination, and heterogeneity between transplant centres will be assessed using bootstrapping, internal-external cross-validation, and methods from meta-analysis.DiscussionThorough evaluation of the existing risk scores for the kidney graft survival or patient-reported outcomes has been lacking in the Swiss transplant setting. In order to be useful in clinical practice, a prognostic score needs to be valid, reliable, clinically relevant, and preferably integrated into the decision-making process to improve long-term patient outcomes and support informed decisions for clinicians and their patients. The state-of-the-art methodology by taking into account competing risks and variable selection using expert knowledge is applied to data from a nationwide prospective multi-center cohort study. Ideally, healthcare providers together with patients can predetermine the risk they are willing to accept from a deceased-donor kidney, with graft survival, quality of life, and graft function estimates available for their consideration.Study registrationOpen Science Framework ID: z6mvj

A comparison of 12 machine learning models developed to predict ploidy, using a morphokinetic meta-dataset of 8147 embryos.

Are machine learning methods superior to traditional statistics in predicting blastocyst ploidy status using morphokinetic and clinical biodata? Mixed effects logistic regression performed better than all machine learning methods for ploidy prediction using our dataset of 8147 embryos. Morphokinetic timings have been demonstrated to be delayed in aneuploid embryos. Machine learning and statistical models are increasingly being built, however, until now they have been limited by data insufficiency. This is a multicentre cohort study. Data were obtained from 8147 biopsied blastocysts from 1725 patients, treated from 2012 to 2020. All embryos were cultured in a time-lapse system at nine IVF clinics in the UK. A total of 3004 euploid embryos and 5023 aneuploid embryos were included in the final verified dataset. We developed a total of 12 models using four different approaches: mixed effects multivariable logistic regression, random forest classifiers, extreme gradient boosting, and deep learning. For each of the four algorithms, two models were created, the first consisting of 22 covariates using 8027 embryos (Dataset 1) and the second, a dataset of 2373 embryos and 26 covariates (Dataset 2). Four final models were created by switching the target outcome from euploid to aneuploid for each algorithm (Dataset 1). Models were validated using internal-external cross-validation and external validation. All morphokinetic variables were significantly delayed in aneuploid embryos. The likelihood of euploidy was significantly increased the more expanded the blastocyst (P < 0.001) and the better the trophectoderm grade (P < 0.01). Univariable analysis showed no association with ploidy status for morula or cleavage stage fragmentation, morula grade, fertilization method, sperm concentration, or progressive motility. Male age did not correlate with the percentage of euploid embryos when stratified for female age. Multinucleation at the two-cell or four-cell stage was not associated with ploidy status. The best-performing model was logistic regression built using the larger dataset with 22 predictors (F1 score 0.59 for predicting euploidy; F1 score 0.77 for predicting aneuploidy; AUC 0.71; 95% CI 0.67-0.73). The best-performing models using the algorithms from random forest, extreme gradient boosting, and deep learning achieved an AUC of 0.68, 0.63, and 0.63, respectively. When using only morphokinetic predictors the AUC was 0.61 for predicting ploidy status, whereas a model incorporating only embryo grading was unable to discriminate aneuploid embryos (AUC = 0.52). The ploidy prediction model's performance improved with increasing age of the egg provider. The models have not been validated in a prospective study design or yet been used to determine whether they improve clinical outcomes. This model may aid decision-making, particularly where pre-implantation genetic testing for aneuploidy is not permitted or for prioritizing embryos for biopsy. No specific funding was sought for this study; university funds supported the first author. A.Ca. is a minor shareholder of participating centres. N/A.

Predicting early extrahepatic recurrence after local treatment of colorectal liver metastases.

Patients who develop early extrahepatic recurrence (EHR) may not benefit from local treatment of colorectal liver metastases (CRLMs). This study aimed to develop a prediction model for early EHR after local treatment of CRLMs using a national data set. A Cox regression prediction model for EHR was developed and validated internally using data on patients who had local treatment for CRLMs with curative intent. Performance assessment included calibration, discrimination, net benefit, and generalizability by internal-external cross-validation. The prognostic relevance of early EHR (within 6 months) was evaluated by landmark analysis. During a median follow-up of 35 months, 557 of the 1077 patients had EHR and 249 died. Median overall survival was 19.5 (95 per cent c.i. 15.6 to 23.0) months in patients with early EHR after CRLM treatment, compared with not reached (45.3 months to not reached) in patients without an early EHR. The EHR prediction model included side and stage of the primary tumour, RAS/BRAFV600E mutational status, and number and size of CRLMs. The range of 6-month EHR predictions was 5.9-56.0 (i.q.r. 12.9-22.0) per cent. The model demonstrated good calibration and discrimination. The C-index through 6 and 12 months was 0.663 (95 per cent c.i. 0.624 to 0.702) and 0.661 (0.632 to 0.689) respectively. The observed 6-month EHR risk was 6.5 per cent for patients in the lowest quartile of predicted risk compared with 32.0 per cent in the highest quartile. Early EHR after local treatment of CRLMs can be predicted.

Development and Validation of Predictive Model for a Diagnosis of First Episode Psychosis Using the Multinational EU-GEI Case–control Study and Modern Statistical Learning Methods

Abstract Background and Hypothesis It is argued that availability of diagnostic models will facilitate a more rapid identification of individuals who are at a higher risk of first episode psychosis (FEP). Therefore, we developed, evaluated, and validated a diagnostic risk estimation model to classify individual with FEP and controls across six countries. Study Design We used data from a large multi-center study encompassing 2627 phenotypically well-defined participants (aged 18–64 years) recruited from six countries spanning 17 research sites, as part of the European Network of National Schizophrenia Networks Studying Gene-Environment Interactions study. To build the diagnostic model and identify which of important factors for estimating an individual risk of FEP, we applied a binary logistic model with regularization by the least absolute shrinkage and selection operator. The model was validated employing the internal-external cross-validation approach. The model performance was assessed with the area under the receiver operating characteristic curve (AUROC), calibration, sensitivity, and specificity. Study Results Having included preselected 22 predictor variables, the model was able to discriminate adults with FEP and controls with high accuracy across all six countries (rangesAUROC = 0.84–0.86). Specificity (range = 73.9–78.0%) and sensitivity (range = 75.6–79.3%) were equally good, cumulatively indicating an excellent model accuracy; though, calibration slope for the diagnostic model showed a presence of some overfitting when applied specifically to participants from France, the UK, and The Netherlands. Conclusions The new FEP model achieved a good discrimination and good calibration across six countries with different ethnic contributions supporting its robustness and good generalizability.

Endovascular versus neurosurgical aneurysm treatment: study protocol for the development and validation of a clinical prediction tool for individualised decision making

IntroductionTreatment decisions for aneurysmal subarachnoid haemorrhage patients should be supported by individualised predictions of the effects of aneurysm treatment. We present a study protocol and analysis plan for the development...

The Safety INdEx of Prehospital On Scene Triage (SINEPOST) study: The development and validation of a risk prediction model to support ambulance clinical transport decisions on-scene.

One of the main problems currently facing the delivery of safe and effective emergency care is excess demand, which causes congestion at different time points in a patient's journey. The modern case-mix of prehospital patients is broad and complex, diverging from the traditional 'time critical accident and emergency' patients. It now includes many low-acuity patients and those with social care and mental health needs. In the ambulance service, transport decisions are the hardest to make and paramedics decide to take more patients to the ED than would have a clinical benefit. As such, this study asked the following research questions: In adult patients attending the ED by ambulance, can prehospital information predict an avoidable attendance? What is the simulated transportability of the model derived from the primary outcome? A linked dataset of 101,522 ambulance service and ED ambulance incidents linked to their respective ED care record from the whole of Yorkshire between 1st July 2019 and 29th February 2020 was used as the sample for this study. A machine learning method known as XGBoost was applied to the data in a novel way called Internal-External Cross Validation (IECV) to build the model. The results showed great discrimination with a C-statistic of 0.81 (95%CI 0.79-0.83) and excellent calibration with an O:E ratio was 0.995 (95% CI 0.97-1.03), with the most important variables being a patient's mobility, their physiological observations and clinical impression with psychiatric problems, allergic reactions, cardiac chest pain, head injury, non-traumatic back pain, and minor cuts and bruising being the most important. This study has successfully developed a decision-support model that can be transformed into a tool that could help paramedics make better transport decisions on scene, known as the SINEPOST model. It is accurate, and spatially validated across multiple geographies including rural, urban, and coastal. It is a fair algorithm that does not discriminate new patients based on their age, gender, ethnicity, or decile of deprivation. It can be embedded into an electronic Patient Care Record system and automatically calculate the probability that a patient will have an avoidable attendance at the ED, if they were transported. This manuscript complies with the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement (Moons KGM, 2015).

Individualised prediction of drug resistance and seizure recurrence after medication withdrawal in people with juvenile myoclonic epilepsy: A systematic review and individual participant data meta-analysis.

A third of people with juvenile myoclonic epilepsy (JME) are drug-resistant. Three-quarters have a seizure relapse when attempting to withdraw anti-seizure medication (ASM) after achieving seizure-freedom. It is currently impossible to predict who is likely to become drug-resistant and safely withdraw treatment. We aimed to identify predictors of drug resistance and seizure recurrence to allow for individualised prediction of treatment outcomes in people with JME. We performed an individual participant data (IPD) meta-analysis based on a systematic search in EMBASE and PubMed - last updated on March 11, 2021 - including prospective and retrospective observational studies reporting on treatment outcomes of people diagnosed with JME and available seizure outcome data after a minimum one-year follow-up. We invited authors to share standardised IPD to identify predictors of drug resistance using multivariable logistic regression. We excluded pseudo-resistant individuals. A subset who attempted to withdraw ASM was included in a multivariable proportional hazards analysis on seizure recurrence after ASM withdrawal. The study was registered at the Open Science Framework (OSF; https://osf.io/b9zjc/). Our search yielded 1641 articles; 53 were eligible, of which the authors of 24 studies agreed to collaborate by sharing IPD. Using data from 2518 people with JME, we found nine independent predictors of drug resistance: three seizure types, psychiatric comorbidities, catamenial epilepsy, epileptiform focality, ethnicity, history of CAE, family history of epilepsy, status epilepticus, and febrile seizures. Internal-external cross-validation of our multivariable model showed an area under the receiver operating characteristic curve of 0·70 (95%CI 0·68-0·72). Recurrence of seizures after ASM withdrawal (n=368) was predicted by an earlier age at the start of withdrawal, shorter seizure-free interval and more currently used ASMs, resulting in an average internal-external cross-validation concordance-statistic of 0·70 (95%CI 0·68-0·73). We were able to predict and validate clinically relevant personalised treatment outcomes for people with JME. Individualised predictions are accessible as nomograms and web-based tools. MING fonds.

Development and validation of a prediction rule for patients suspected of acute coronary syndrome in primary care: a cross-sectional study

ObjectiveTo develop and validate a symptom-based prediction rule for early recognition of acute coronary syndrome (ACS) in patients with acute chest discomfort who call out-of-hours services for primary care (OHS-PC).DesignCross-sectional...

Accurately Identifying Cerebroarterial Stenosis from Angiography Reports Using Natural Language Processing Approaches.

Patients with intracranial artery stenosis show high incidence of stroke. Angiography reports contain rich but underutilized information that can enable the detection of cerebrovascular diseases. This study evaluated various natural language processing (NLP) techniques to accurately identify eleven intracranial artery stenosis from angiography reports. Three NLP models, including a rule-based model, a recurrent neural network (RNN), and a contextualized language model, XLNet, were developed and evaluated by internal–external cross-validation. In this study, angiography reports from two independent medical centers (9614 for training and internal validation testing and 315 as external validation) were assessed. The internal testing results showed that XLNet had the best performance, with a receiver operating characteristic curve (AUROC) ranging from 0.97 to 0.99 using eleven targeted arteries. The rule-based model attained an AUROC from 0.92 to 0.96, and the RNN long short-term memory model attained an AUROC from 0.95 to 0.97. The study showed the potential application of NLP techniques such as the XLNet model for the routine and automatic screening of patients with high risk of intracranial artery stenosis using angiography reports. However, the NLP models were investigated based on relatively small sample sizes with very different report writing styles and a prevalence of stenosis case distributions, revealing challenges for model generalization.

Personalizing treatment in end-stage kidney disease: deciding between haemodiafiltration and haemodialysis based on individualized treatment effect prediction.

ABSTRACTBackgroundPrevious studies suggest that haemodiafiltration reduces mortality compared with haemodialysis in patients with end-stage kidney disease (ESKD), but the controversy surrounding its benefits remains and it is unclear to what extent individual patients benefit from haemodiafiltration. This study is aimed to develop and validate a treatment effect prediction model to determine which patients would benefit most from haemodiafiltration compared with haemodialysis in terms of all-cause mortality.MethodsIndividual participant data from four randomized controlled trials comparing haemodiafiltration with haemodialysis on mortality were used to derive a Royston-Parmar model for the prediction of absolute treatment effect of haemodiafiltration based on pre-specified patient and disease characteristics. Validation of the model was performed using internal-external cross validation.ResultsThe median predicted survival benefit was 44 (Q1–Q3: 44–46) days for every year of treatment with haemodiafiltration compared with haemodialysis. The median survival benefit with haemodiafiltration ranged from 2 to 48 months. Patients who benefitted most from haemodiafiltration were younger, less likely to have diabetes or a cardiovascular history and had higher serum creatinine and albumin levels. Internal–external cross validation showed adequate discrimination and calibration.ConclusionAlthough overall mortality is reduced by haemodiafiltration compared with haemodialysis in ESKD patients, the absolute survival benefit can vary greatly between individuals. Our results indicate that the effects of haemodiafiltration on survival can be predicted using a combination of readily available patient and disease characteristics, which could guide shared decision-making.

Personalized Model to Predict Keratoconus Progression From Demographic, Topographic, and Genetic Data

To generate a prognostic model to predict keratoconus progression to corneal crosslinking (CXL). Retrospective cohort study. We recruited 5025 patients (9341 eyes) with early keratoconus between January 2011 and November 2020. Genetic data from 926 patients were available. We investigated both keratometry or CXL as end points for progression and used the Royston-Parmar method on the proportional hazards scale to generate a prognostic model. We calculated hazard ratios (HRs) for each significant covariate, with explained variation and discrimination, and performed internal-external cross validation by geographic regions. After exclusions, model fitting comprised 8701 eyes, of which 3232 underwent CXL. For early keratoconus, CXL provided a more robust prognostic model than keratometric progression. The final model explained 33% of the variation in time to event: age HR (95% CI) 0.9 (0.90-0.91), maximum anterior keratometry 1.08 (1.07-1.09), and minimum corneal thickness 0.95 (0.93-0.96) as significant covariates. Single-nucleotide polymorphisms (SNPs) associated with keratoconus (n=28) did not significantly contribute to the model. The predicted time-to-event curves closely followed the observed curves during internal-external validation. Differences in discrimination between geographic regions was low, suggesting the model maintained its predictive ability. A prognostic model to predict keratoconus progression could aid patient empowerment, triage, and service provision. Age at presentation is the most significant predictor of progression risk. Candidate SNPs associated with keratoconus do not contribute to progression risk.

Development of the ADFICE_IT Models for Predicting Falls and Recurrent Falls in Community-Dwelling Older Adults: Pooled Analyses of European Cohorts With Special Attention to Medication.

BackgroundUse of fall prevention strategies requires detection of high-risk patients. Our goal was to develop prediction models for falls and recurrent falls in community-dwelling older adults and to improve upon previous models by using a large, pooled sample and by considering a wide range of candidate predictors, including medications.MethodsHarmonized data from 2 Dutch (LASA, B-PROOF) and 1 German cohort (ActiFE Ulm) of adults aged ≥65 years were used to fit 2 logistic regression models: one for predicting any fall and another for predicting recurrent falls over 1 year. Model generalizability was assessed using internal–external cross-validation.ResultsData of 5 722 participants were included in the analyses, of whom 1 868 (34.7%) endured at least 1 fall and 702 (13.8%) endured a recurrent fall. Positive predictors for any fall were: educational status, depression, verbal fluency, functional limitations, falls history, and use of antiepileptics and drugs for urinary frequency and incontinence; negative predictors were: body mass index (BMI), grip strength, systolic blood pressure, and smoking. Positive predictors for recurrent falls were: educational status, visual impairment, functional limitations, urinary incontinence, falls history, and use of anti-Parkinson drugs, antihistamines, and drugs for urinary frequency and incontinence; BMI was a negative predictor. The average C-statistic value was 0.65 for the model for any fall and 0.70 for the model for recurrent falls.ConclusionCompared with previous models, the model for recurrent falls performed favorably while the model for any fall performed similarly. Validation and optimization of the models in other populations are warranted.

Development and validation of clinical prediction models for breast cancer incidence and mortality: a protocol for a dual cohort study.

IntroductionBreast cancer is the most common cancer and the leading cause of cancer-related death in women worldwide. Risk prediction models may be useful to guide risk-reducing interventions (such as pharmacological agents) in women at increased risk or inform screening strategies for early detection methods such as screening.Methods and analysisThe study will use data for women aged 20–90 years between 2000 and 2020 from QResearch linked at the individual level to hospital episodes, cancer registry and death registry data. It will evaluate a set of modelling approaches to predict the risk of developing breast cancer within the next 10 years, the ‘combined’ risk of developing a breast cancer and then dying from it within 10 years, and the risk of breast cancer mortality within 10 years of diagnosis. Cox proportional hazards, competing risks, random survival forest, deep learning and XGBoost models will be explored. Models will be developed on the entire dataset, with ‘apparent’ performance reported, and internal-external cross-validation used to assess performance and geographical and temporal transportability (two 10-year time periods). Random effects meta-analysis will pool discrimination and calibration metric estimates from individual geographical units obtained from internal-external cross-validation. We will then externally validate the models in an independent dataset. Evaluation of performance heterogeneity will be conducted throughout, such as exploring performance across ethnic groups.Ethics and disseminationEthics approval was granted by the QResearch scientific committee (reference number REC 18/EM/0400: OX129). The results will be written up for submission to peer-reviewed journals.