Calibration Slope Research Articles

Understanding overfitting in random forest for probability estimation: a visualization and simulation study

BackgroundRandom forests have become popular for clinical risk prediction modeling. In a case study on predicting ovarian malignancy, we observed training AUCs close to 1. Although this suggests overfitting, performance was competitive on test data. We aimed to understand the behavior of random forests for probability estimation by (1) visualizing data space in three real-world case studies and (2) a simulation study.MethodsFor the case studies, multinomial risk estimates were visualized using heatmaps in a 2-dimensional subspace. The simulation study included 48 logistic data-generating mechanisms (DGM), varying the predictor distribution, the number of predictors, the correlation between predictors, the true AUC, and the strength of true predictors. For each DGM, 1000 training datasets of size 200 or 4000 with binary outcomes were simulated, and random forest models were trained with minimum node size 2 or 20 using the ranger R package, resulting in 192 scenarios in total. Model performance was evaluated on large test datasets (N = 100,000).ResultsThe visualizations suggested that the model learned “spikes of probability” around events in the training set. A cluster of events created a bigger peak or plateau (signal), isolated events local peaks (noise). In the simulation study, median training AUCs were between 0.97 and 1 unless there were 4 binary predictors or 16 binary predictors with a minimum node size of 20. The median discrimination loss, i.e., the difference between the median test AUC and the true AUC, was 0.025 (range 0.00 to 0.13). Median training AUCs had Spearman correlations of around 0.70 with discrimination loss. Median test AUCs were higher with higher events per variable, higher minimum node size, and binary predictors. Median training calibration slopes were always above 1 and were not correlated with median test slopes across scenarios (Spearman correlation − 0.11). Median test slopes were higher with higher true AUC, higher minimum node size, and higher sample size.ConclusionsRandom forests learn local probability peaks that often yield near perfect training AUCs without strongly affecting AUCs on test data. When the aim is probability estimation, the simulation results go against the common recommendation to use fully grown trees in random forest models.

SACrA score to predict the initiation of renal replacement therapy in critically ill patients: a single-center retrospective study

Background Acute kidney injury (AKI) is a prevalent complication in critically ill patients that affects the timing of renal replacement therapy (RRT) initiation. This study aimed to develop and validate the SACrA score for predicting non-emergent initiations (BUN ≥112 mg/dL or oliguria for >72 h) of RRT in critically ill patients. Methods We conducted a retrospective cohort study using data from two cohorts. The derivation cohort included patients admitted to the ICU between November 2021 and December 2023, whereas the validation cohort included patients admitted between September 2019 and October 2021. The primary outcome was non-emergent RRT initiation. The multivariate logistic regression with stepwise selection based on the Akaike information criterion finalized the model, including the variables, such as sex, albumin (Alb), creatinine (Cr), and APACHE II score (SACrA). Results The derivation and validation cohorts comprised 470 and 476 patients, respectively. The SACrA score showed a strong predictive performance for non-emergent RRT initiation in both the cohorts. Cohort 1 had an ROC–AUC of 0.971, with a calibration slope of 0.982 and an intercept of 0.009, whereas cohort 2 had an ROC–AUC of 0.918, with a calibration slope of 0.988 and an intercept of 0.004. Conclusions The SACrA score is a robust tool for predicting non-emergent RRT initiation in critically ill patients using readily available clinical variables. Though additional data are needed to validate the SACrA score, our analysis suggests the tool may help clinicians make informed decisions, reduce unnecessary RRT, and thereby improve patient outcomes.

Predicting extended hospital stay following revision total hip arthroplasty: a machine learning model analysis based on the ACS-NSQIP database.

Prolonged length of stay (LOS) following revision total hip arthroplasty (THA) can lead to increased healthcare costs, higher rates of readmission, and lower patient satisfaction. In this study, we investigated the predictive power of machine learning (ML) models for prolonged LOS after revision THA using patient data from a national-scale patient repository. We identified 11,737 revision THA cases from the American College of Surgeons National Surgical Quality Improvement Program database from 2013 to 2020. Prolonged LOS was defined as exceeding the 75th value of all LOSs in the study cohort. We developed four ML models: artificial neural network (ANN), random forest, histogram-based gradient boosting, and k-nearest neighbor, to predict prolonged LOS after revision THA. Each model's performance was assessed during training and testing sessions in terms of discrimination, calibration, and clinical utility. The ANN model was the most accurate with an AUC of 0.82, calibration slope of 0.90, calibration intercept of 0.02, and Brier score of 0.140 during testing, indicating the model's competency in distinguishing patients subject to prolonged LOS with minimal prediction error. All models showed clinical utility by producing net benefits in the decision curve analyses. The most significant predictors of prolonged LOS were preoperative blood tests (hematocrit, platelet count, and leukocyte count), preoperative transfusion, operation time, indications for revision THA (infection), and age. Our study demonstrated that the ML model accurately predicted prolonged LOS after revision THA. The results highlighted the importance of the indications for revision surgery in determining the risk of prolonged LOS. With the model's aid, clinicians can stratify individual patients based on key factors, improve care coordination and discharge planning for those at risk of prolonged LOS, and increase cost efficiency.

Validation of the BOADICEA model for epithelial tubo-ovarian cancer risk prediction in UK Biobank

BackgroundThe clinical validity of the multifactorial BOADICEA model for epithelial tubo-ovarian cancer (EOC) risk prediction has not been assessed in a large sample size or over a longer term.MethodsWe evaluated the model discrimination and calibration in the UK Biobank cohort comprising 199,429 women (733 incident EOCs) of European ancestry without previous cancer history. We predicted 10-year EOC risk incorporating data on questionnaire-based risk factors (QRFs), family history, a 36-SNP polygenic risk score and pathogenic variants (PV) in six EOC susceptibility genes (BRCA1, BRCA2, RAD51C, RAD51D, BRIP1 and PALB2).ResultsDiscriminative ability was maximised under the multifactorial model that included all risk factors (AUC = 0.68, 95% CI: 0.66–0.70). This model was well calibrated in deciles of predicted risk with calibration slope=0.99 (95% CI: 0.98–1.01). Discriminative ability was similar in women younger or older than 60 years. The AUC was higher when analyses were restricted to PV carriers (0.76, 95% CI: 0.69–0.82). Using relative risk (RR) thresholds, the full model classified 97.7%, 1.7%, 0.4% and 0.2% women in the RR < 2.0, 2.0 ≤ RR < 2.9, 2.9 ≤ RR < 6.0 and RR ≥ 6.0 categories, respectively, identifying 9.1 of incident EOC among those with RR ≥ 2.0.DiscussionBOADICEA, implemented in CanRisk (www.canrisk.org), provides valid 10-year EOC risks and can facilitate clinical decision-making in EOC risk management.

External validation of the BE-ALIVE score for predicting 30-day mortality in patients presenting with acute coronary syndromes

IntroductionThe BE-ALIVE score is an additive scoring system for estimating 30-day mortality in patients presenting with an acute coronary syndrome (ACS) [1]. However, it had only previously been tested on an internal validation cohort. The aim was to assess the scoring system on an external validation cohort. MethodsThe scoring system comprises six domains: (1) Base Excess (1 point for < −2 mmols/L), (2) Age (<65 years: 0 points, 65–74: 1 point, 75–84: 2 points, ≥ 85: 3 points), (3) Lactate (<2 mmols/L: 0 points, 2–4.9: 1 point, 5–9.9: 3 points, ≥ 10: 6 points), (4) Intubated & Ventilated (2 points), (5) Left Ventricular function (normal or mildly impaired: -1 point, moderately impaired: 1 point, severely impaired: 3 points) and (6) External / out of hospital cardiac arrest (1 point).We applied the BE-ALIVE score was applied to 205 consecutive patients at a different institution. ResultsCalibration was strong, with an observed to expected ratio of 1.01, a calibration slope of 1.26 and calibration in the large of −0.03. The Spiegelhalter's Z-statistic was −0.95 (p = 0.34). The AUC was 0.95 (0.92–0.98) in the external validation cohort versus 0.90 (0.85–0.95) during internal validation.Overall performance was excellent with a Brier score of 0.07 versus 0.06 during internal validation. The negative predictive value for 30-day mortality of a BE-ALIVE score < 4 was 98 %, with a positive predicted value of a score ≥ 10 of 95 %. ConclusionsThe BE-ALIVE score remains a robust predictor of 30-day mortality in an external validation cohort.

Risk-scoring system predicting need for hospital-specific interventional care after peroral endoscopic myotomy.

Early identification of patients needing hospital-specific interventional care (HIC) following endoscopic treatment is valuable for optimizing postoperative hospital stays. We aimed to develop and validate a risk-scoring system for predicting HIC in patients who underwent peroral endoscopic myotomy (POEM). This study included patients with esophageal motility disorders who underwent POEM at our hospital between April 2015 and March 2023. HIC was defined as any of the following situations: fasting for gastrointestinal rest to manage adverse events (AEs); intravenous administration of medications such as antibiotics and blood transfusion; endoscopic, radiologic, and surgical interventions; intensive care unit management; or other life-threatening events. A risk-scoring system for predicting HIC after postoperative day (POD) 1 was developed using multivariable logistic regression and was internally validated using bootstrapping and decision curve analysis. Of the 589 patients, 50 (8.5%) experienced HIC after POD1. Risk scores were assigned for four factors as follows: age (0 points for <70 years, 1 point for 70-79 years, 2 points for ≥80 years), preoperative prognostic nutritional index (0 points for >45, 1 point for 40-45, 4 points for <40), postoperative surgical site AEs on second-look endoscopy (7 points), and postoperative pneumonia on chest radiography (6 points). The discriminative ability (concordance statistics, 0.85; 95% confidence interval, 0.78-0.91) and calibration (slope 1.00; 0.74-1.28) were satisfactory. The decision curve analysis demonstrated its clinical usefulness. This risk-scoring system can predict the HIC after POD1 and provide useful information for determining discharge.

A Multicenter Validation of a Novel Prediction Model for Elbow Flexion Recovery after Nerve Transfer Surgery in Brachial Plexus Injuries.

Nerve transfer surgery for brachial plexus injuries exhibits variable success rates, potentially resulting in prolonged limb dysfunction for more than 2 years. A proposed prediction model has been developed to predict the unsuccessful recovery of elbow flexion after the surgery. The model consisted of six variables, namely body mass index 23 kg/m2 or more, smoking, total arm type, donor nerve, ipsilateral upper extremity fracture, and ipsilateral vascular injury. This study aimed to assess the external validity of the model for wider applicability. This retrospective analysis examined the medical records of 213 eligible patients with traumatic brachial plexus injuries who underwent surgery at two referral centers between July 2008 and June 2022. The prediction model was applied to estimate recovery failure probability, which was compared with the observed outcomes for each patient. Both the original and simplified models were validated for discrimination and calibration using metrics including c-statistic, Hosmer-Lemeshow goodness-of-fit test, calibration plot, calibration slope, and intercept. Thirty-two percent of patients experienced unsuccessful elbow flexion recovery. Both the original and simplified models demonstrated good discrimination (c-statistics: 0.748 and 0.759, respectively). The Hosmer-Lemeshow test revealed strong agreement between predicted and observed probabilities for both models (P = 0.66 and P = 0.92, respectively). The calibration plot exhibited good agreement, with a calibration slope of 0.928 and an intercept of 0.377. The prediction model showed strong external validation, confirming its clinical value. High-risk patients should be educated on the risks and benefits of nerve transfer surgery and consider alternative treatments such as primary free functioning muscle transfer.

The performance of prognostic models depended on the choice of missing value imputation algorithm: a simulation study

ObjectiveThe development of clinical prediction models is often impeded by the occurrence of missing values in the predictors. Various methods for imputing missing values before modelling have been proposed. Some of them are based on variants of multiple imputation by chained equations, while others are based on single imputation. These methods may include elements of flexible modelling or machine learning algorithms, and for some of them user-friendly software packages are available. The aim of this study was to investigate by simulation if some of these methods consistently outperform others in performance measures of clinical prediction models. Study Design and SettingWe simulated development and validation cohorts by mimicking observed distributions of predictors and outcome variable of a real data set. In the development cohorts, missing predictor values were created in 36 scenarios defined by the missingness mechanism and proportion of non-complete cases. We applied three imputation algorithms that were available in R software: mice, aregImpute and missForest. These algorithms differed in their use of linear or flexible models, or random forests, the way of sampling from the predictive posterior distribution, and the generation of a single or multiple imputed data sets. For multiple imputation we also investigated the impact of the number of imputations. Logistic regression models were fitted with the simulated development cohorts before (full data analysis) and after missing value generation (complete case analysis), and with the imputed data. Prognostic model performance was measured by the scaled Brier score, c-statistic, calibration intercept and slope, and by the mean absolute prediction error evaluated in validation cohorts without missing values. Performance of full data analysis was considered as ideal. ResultsNone of the imputation methods achieved the model’s predictive accuracy that would be obtained in case of no missingness. In general, complete case analysis yielded the worst performance, and deviation from ideal performance increased with increasing percentage of missingness and decreasing sample size. Across all scenarios and performance measures, aregImpute and mice, both with 100 imputations, resulted in highest predictive accuracy. Surprisingly aregImpute outperformed full data analysis in achieving calibration slopes very close to 1 across all scenarios and outcome models. The increase of mice’s performance with 100 compared to 5 imputations was only marginal. The differences between the imputation methods decreased with increasing sample sizes and decreasing proportion of non-complete cases. ConclusionIn our simulation study, model calibration was more affected by the choice of the imputation method than model discrimination. While differences in model performance after using imputation methods were generally small, multiple imputation methods as mice and aregImpute that can handle linear or nonlinear associations between predictors and outcome are an attractive and reliable choice in most situations.

Factors related to the occurrence of fetal birth defects and the construction of a Nomogram model.

To explore the influencing factors of fetal birth defects (BD) and construct a nomogram model. A total of 341 newborns admitted to Meizhou people's hospital from September 2021 to September 2023 were randomly grouped into a modeling group (239 cases) and a validation group (102 cases). The modeling group fetuses were separated into BD and non-BD groups. Multivariate logistic regression analyzed risk factors for BD; R software constructed a nomogram model; Receiver operating characteristic (ROC) curve evaluated the model's discrimination for BD. The top 5 types of BD were congenital heart disease, polydactyly/syndactyly, cleft lip/palate, ear malformation, and foot malformation, with incidence rates of 23.81%, 20.63%, 12.70%, 11.11%, and 7.94%, respectively. BD incidence was 26.36% (63/239). Significant differences between BD and non-BD groups were found in maternal age, gestational age, history of adverse pregnancy/childbirth, gestational hypertension, adverse emotions during pregnancy, and folic acid intake duration (P<0.05). Logistic regression showed maternal age (OR: 4.125), gestational age (OR: 3.066), adverse pregnancy history (OR: 10.628), gestational hypertension (OR: 5.658), adverse emotions (OR: 5.467), and folic acid intake duration (OR: 4.586) were risk factors for BD (P<0.05). The modeling group's ROC AUC was 0.938, calibration curve slope close to 1, H-L test =8.342, P=0.692; external validation AUC was 0.961, calibration slope close to 1, H-L test =7.634, P=0.635. Identified risk factors include maternal age, gestational age, adverse pregnancy history, gestational hypertension, adverse emotions, and folic acid intake duration. The nomogram model shows good discrimination and consistency for evaluating neonatal BD risk.

Long-Term Outcomes of Patients With Apical Hypertrophic Cardiomyopathy Utilizing a New Risk Score

BackgroundApical hypertrophic cardiomyopathy (aHCM) is a distinct variant characterized by predominant hypertrophy of the left ventricle apex. ObjectivesThis study sought to describe aHCM patients' characteristics and develop a risk score for aHCM patients. MethodsA total of 462 patients (age 58 ± 15 years, 68% male) diagnosed with aHCM were included. The primary endpoint was death, appropriate defibrillator discharge, or need for cardiac transplantation. Variables showing potential association with the composite endpoint were considered to develop an aHCM-specific risk score. ResultsAt baseline, 67% patients were asymptomatic and 69% had no risk factors for sudden death. On echocardiography, the mean left ventricle ejection fraction, left atrial volume index, and right ventricular systolic pressure were 64% ± 8%, 36 ± 15 ml/m2, and 32 ± 10 mm Hg, respectively, with 51(11%) demonstrating an apical aneurysm. Baseline cardiac magnetic resonance, performed in 246 (53%) patients, demonstrated delayed gadolinium enhancement in 170 (71%) patients (mean percentage of 4.9% ± 6.6%). At age 6.3 ± 4.8 years, the composite events occurred in 80 (17%, death in 62 [13%]) patients. The aHCM-specific risk score, incorporating age, apical aneurysm, left atrial volume index, serum creatinine, and right ventricular systolic pressure, demonstrated good discrimination (C-statistic = 0.75) with an expected to observed ratio of 1.02 and a calibration slope of 0.91. The risk score ranged between 0 and 8 points, with a higher score associated with higher composite events. ConclusionsaHCM constituted 6.8% of our overall HCM cohort with a composite event rate of 2.8%/year. The aHCM risk score provided good discrimination in predicting the composite primary endpoint, with a higher score associated with a higher rate of events.

The PROgnostic ModEl for chronic lung disease (PRO-MEL): development and temporal validation

BackgroundPatients with chronic lung diseases (CLDs), defined as progressive and life-limiting respiratory conditions, experience a heavy symptom burden as the conditions become more advanced, but palliative referral rates are low and late. Prognostic tools can help clinicians identify CLD patients at high risk of deterioration for needs assessments and referral to palliative care. As current prognostic tools may not generalize well across all CLD conditions, we aim to develop and validate a general model to predict one-year mortality in patients presenting with any CLD.MethodsA retrospective cohort study of patients with a CLD diagnosis at a public hospital from July 2016 to October 2017 was conducted. The outcome of interest was all-cause mortality within one-year of diagnosis. Potential prognostic factors were identified from reviews of prognostic studies in CLD, and data was extracted from electronic medical records. Missing data was imputed using multiple imputation by chained equations. Logistic regression models were developed using variable selection methods and validated in patients seen from January 2018 to December 2019. Discriminative ability, calibration and clinical usefulness of the model was assessed. Model coefficients and performance were pooled across all imputed datasets and reported.ResultsOf the 1000 patients, 122 (12.2%) died within one year. Patients had chronic obstructive pulmonary disease or emphysema (55%), bronchiectasis (38%), interstitial lung diseases (12%), or multiple diagnoses (6%). The model selected through forward stepwise variable selection had the highest AUC (0.77 (0.72–0.82)) and consisted of ten prognostic factors. The model AUC for the validation cohort was 0.75 (0.70, 0.81), and the calibration intercept and slope were − 0.14 (-0.54, 0.26) and 0.74 (0.53, 0.95) respectively. Classifying patients with a predicted risk of death exceeding 0.30 as high risk, the model would correctly identify 3 out 10 decedents and 9 of 10 survivors.ConclusionsWe developed and validated a prognostic model for one-year mortality in patients with CLD using routinely available administrative data. The model will support clinicians in identifying patients across various CLD etiologies who are at risk of deterioration for a basic palliative care assessment to identify unmet needs and trigger an early referral to palliative medicine.Trial registrationNot applicable (retrospective study).

Development and Internal-external Validation of a Post-operative Mortality Risk Calculator for Pediatric Surgical Patients in Low- and Middle- Income Countries Using Machine Learning

BackgroundThe purpose of this study was to develop and validate a mortality risk algorithm for pediatric surgery patients treated at KidsOR sites in 14 low- and middle-income countries. MethodsA SuperLearner machine learning algorithm was trained to predict post-operative mortality by hospital discharge using the retrospectively and prospectively collected KidsOR database including patients treated at 20 KidsOR sites from June 2018 to June 2023. Algorithm performance was evaluated by internal-external cross-validated AUC and calibration. FindingsOf 23,905 eligible patients, 21,703 with discharge status recorded were included in the analysis, representing a post-operative mortality rate of 3.1% (671 mortality events). The candidate algorithm with the best cross-validated performance was an extreme gradient boosting model. The cross-validated AUC was 0.945 (95% CI 0.936 to 0.954) and cross-validated calibration slope and intercept were 1.01 (95% CI 0.96 to 1.06) and 0.05 (95% CI -0.10 to 0.21). For Super Learner models trained on all but one site and evaluated in the holdout site for sites with at least 25 mortality events, overall external validation AUC was 0.864 (95% CI 0.846 to 0.882) with calibration slope and intercept of 1.03 (95% CI 0.97 to 1.09) and 1.18 (95% CI 0.98 to 1.39). InterpretationThe KidsOR post-operative mortality risk algorithm had outstanding cross-validated discrimination and strong cross-validated calibration. Across all external validation sites, discrimination of Super Learner models trained on the remaining sites was excellent, though re-calibration may be necessary prior to use at new sites. This model has the potential to inform clinical practice and guide resource allocation at KidsOR sites world-wide. Type of Study and Level of EvidenceObservational Study, Level III.

Construction and evaluation of a mortality prediction model for patients with acute kidney injury undergoing continuous renal replacement therapy based on machine learning algorithms

Background To construct and evaluate a predictive model for in-hospital mortality among critically ill patients with acute kidney injury (AKI) undergoing continuous renal replacement therapy (CRRT), based on nine machine learning (ML) algorithm. Methods The study retrospectively included patients with AKI who underwent CRRT during their initial hospitalization in the United States using the medical information mart for intensive care (MIMIC) database IV (version 2.0), as well as in the intensive care unit (ICU) of Huzhou Central Hospital. Patients from the MIMIC database were used as the training cohort to construct the models (from 2008 to 2019, n = 1068). Patients from Huzhou Central Hospital were utilized as the external validation cohort to evaluate the models (from June 2019 to December 2022, n = 327). In the training cohort, least absolute shrinkage and selection operator (LASSO) regression with cross-validation was employed to select features for constructing the model and subsequently established nine ML predictive models. The performance of these nine models on the external validation cohort dataset was comprehensively evaluated based on the area under the receiver operating characteristic curve (AUROC) and the optimal model was selected. A static nomogram and a web-based dynamic nomogram were presented, with a comprehensive evaluation from the perspectives of discrimination (AUROC), calibration (calibration curve) and clinical practicability (DCA curves). Results Finally, 1395 eligible patients were enrolled, including 1068 patients in the training cohort and 327 patients in the external validation cohort. In the training cohort, LASSO regression with cross-validation was employed to select features and nine models were individually constructed. Compared to the other eight models, the Lasso regularized logistic regression (Lasso-LR) model exhibited the highest AUROC (0.756) and the optimal calibration curve. The DCA curve suggested a certain clinical utility in predicting in-hospital mortality among critically ill patients with AKI undergoing CRRT. Consequently, the Lasso-LR model was the optimal model and it was visualized as a common nomogram (static nomogram) and a web-based dynamic nomogram (https://chsyh2006.shinyapps.io/dynnomapp/). Discrimination, calibration and DCA curves were employed to assess the performance of the nomogram. The AUROC for the training and external validation cohorts in the nomogram model was 0.771 (95%CI: 0.743, 0.799) and 0.756 (95%CI: 0.702, 0.809), respectively. The calibration slope and Brier score for the training cohort were 1.000 and 0.195, while for the external validation cohort, they were 0.849 and 0.197, respectively. The DCA indicated that the model had a certain clinical application value. Conclusions Our study selected the optimal model and visualized it as a static and dynamic nomogram integrating clinical predictors, so that clinicians can personalized predict the in-hospital outcome of critically ill patients with AKI undergoing CRRT upon ICU admission.

Individualized MRI-based stroke PRediction scOre using plaque Vulnerability for symptomatic carotid artEry disease patients (IMPROVE).

In TIA and stroke patients with carotid stenosis, estimations of future ipsilateral ischemic stroke risk and treatment decisions are currently primarily based on the degree of stenosis. Intraplaque hemorrhage (IPH), which can be readily visualized on carotid MRI, is increasingly established as an easy to assess and a very strong and independent predictor for ipsilateral stroke risk, stronger than any clinical risk factor. We developed a clinical prediction model (IMPROVE) incorporating IPH, degree of stenosis, and clinical risk factors to select patients with symptomatic carotid stenosis at high risk for stroke. IMPROVE was developed on pooled clinical and MRI data from five cohort studies of 760 recent TIA or minor stroke patients with carotid plaque who received optimal medical treatment. We used Cox proportional hazards models to determine the coefficients of IMPROVE. IMPROVE was internally validated using bootstrapping and converted to one- and three-year ipsilateral ischemic stroke risk. The development dataset contained 65 ipsilateral incident ischemic strokes that occurred during a median follow-up of 1.2 years (IQR: 0.5-4.1). The IMPROVE model includes five predictors, which are in order of importance: degree of stenosis, presence of IPH on MRI, classification of last event (cerebral vs ocular), sex, and age. Internal validation revealed a good accuracy (C-statistic: 0.82; 95% CI: 0.77-0.87) and no evidence for miscalibration (calibration slope: 0.93). Using presence of IPH on MRI and only four conventional parameters, the IMPROVE model provides accurate individual stroke risk estimates, which may facilitate stratification for revascularization.

Development and external validation of a model to predict multidrug-resistant bacterial infections in patients with cirrhosis.

We included patients with cirrhosis and bacterial infections from two prospective studies: a transcontinental study was used for model development and internal validation (n = 1302), and a study from Argentina and Uruguay was used for external validation (n = 472). All predictors were measured at the time of infection. Both culture-positive and culture-negative infections were included. The model was developed using logistic regression with backward stepwise predictor selection. We externally validated the optimism-adjusted model using calibration and discrimination statistics and evaluated its clinical utility. The prevalence of MDRO infections was 19% and 22% in the development and external validation datasets, respectively. The model's predictors were sex, prior antibiotic use, type and site of infection, MELD-Na, use of vasopressors, acute-on-chronic liver failure, and interaction terms. Upon external validation, the calibration slope was 77 (95% CI .48-1.05), and the area under the ROC curve was .68 (95% CI .61-.73). The application of the model significantly changed the post-test probability of having an MDRO infection, identifying patients with nosocomial infection at very low risk (8%) and patients with community-acquired infections at significant risk (36%). This model achieved adequate performance and could be used to improve the selection of empiric antibiotics, aligning with other antibiotic stewardship program strategies.

Development and validation of a prognostic model to predict birth weight: individual participant data meta-analysis

ObjectiveTo predict birth weight at various potential gestational ages of delivery based on data routinely available at the first antenatal visit.DesignIndividual participant data meta-analysis.Data sourcesIndividual participant data of four cohorts...

External validation of models to estimate gestational age in the second and third trimester using ultrasound: A prospective multicentre observational study.

Accurate assessment of gestational age (GA) is important at both individual and population levels. The most accurate way to estimate GA in women who book late in pregnancy is unknown. The aim of this study was to externally validate the accuracy of equations for GA estimation in late pregnancy and to identify the best equation for estimating GA in women who do not receive an ultrasound scan until the second or third trimester. This was a prospective, observational cross-sectional study. 57 prenatal care centres, France. Women with a singleton pregnancy and a previous 11-14-week dating scan that gave the observed GA were recruited over an 8-week period. They underwent a standardised ultrasound examination at one time point during the pregnancy (15-43 weeks), measuring 12 foetal biometric parameters that have previously been identified as useful for GA estimation. A total of 189 equations that estimate GA based on foetal biometry were examined and compared with GA estimation based on foetal CRL. Comparisons between the observed GA and the estimated GA were made using R2, calibration slope and intercept. RMSE, mean difference and 95% range of error were also calculated. A total of 2741 pregnant women were examined. After exclusions, 2339 participants were included. In the 20 best performing equations, the intercept ranged from -0.22 to 0.30, the calibration slope from 0.96 to 1.03 and the RSME from 0.67 to 0.87. Overall, multiparameter models outperformed single-parameter models. Both the 95% range of error and mean difference increased with gestation. Commonly used models based on measurement of the head circumference alone were not amongst the best performing models and were associated with higher 95% error and mean difference. We provide strong evidence that GA-specific equations based on multiparameter models should be used to estimate GA in late pregnancy. However, as all methods of GA assessment in late pregnancy are associated with large prediction intervals, efforts to improve access to early antenatal ultrasound must remain a priority. The proposal for this study and the corresponding methodological review was registered on PROSPERO international register of systematic reviews (registration number: CRD4201913776).

Supporting vaccine (co)-administration decisions: Development and validation of a tool for assessing the risk of severe outcomes due to lower respiratory tract infections

ObjectiveTo develop and validate a score to predict the 90-day risk of hospitalization/death in patients with low respiratory tract infections (LRTIs) with the aim to support clinical decision making on vaccine (co)-administration. MethodsWe formed a cohort of patients aged 18 years or older being diagnosed with LRTIs in the period between January 1, 2012 and December 31, 2022. Each patient was followed until occurrence of respiratory-related hospitalization/death up to the end of the study period (December 31, 2022). Along with age and sex, forty determinants were adopted to assemble the respiratory tract infection (RTI)-Health Search (HS) core using the development sub-cohort. The prediction accuracy of the score was therefore assessed in the validation sub-cohort. ResultsWe identified 252,319 patients being diagnosed with LRTIs (females: 54.7 %; mean age: 60 (SD:18.1)). When the risk of LRTIs-related hospitalizations/deaths was estimated via RTI-HScore, its predicted value was equal to 1.4 % over a 90-day event horizon. The score showed explained variation and discrimination accuracy were equal to 45 % (95 % CI: 44–47 %) and 81 % (95 % CI: 79–84 %), respectively. The calibration slope did not significantly differ from the unit (p = 0.8314). ConclusionsThe RTI-HScore was featured by good accuracy for prediction of LRTIs-related complications over a 90-day follow-up. Such a tool might therefore support general practitioners to enhance patients’ care by facilitating approaches for (co)-administration of vaccines for respiratory infections through a score-based decision support system.

Matrix effects in ultra-high performance supercritical fluid chromatography-mass spectrometry analysis of vitamin E in plasma: The effect of sample preparation and data processing

The approaches to matrix effects determination and reduction in ultra-high performance supercritical fluid chromatography with mass spectrometry detection have been evaluated in this study using different sample preparation methods and investigation of different calibration models. Five sample preparation methods, including protein precipitation, liquid-liquid extraction, supported liquid extraction, and solid phase extraction based on both “bind and elute” and “interferent removal” modes, were optimized with an emphasis on the matrix effects and recovery of 8 forms of vitamin E, including α-, β-, γ-, and δ-tocopherols and tocotrienols, from plasma. The matrix effect evaluation included the use and comparison of external and internal calibration using three models, i.e., least square with no transformation and no weighting (1/x0), with 1/x2 weighting, and with logarithmic transformation. The calibration model with logarithmic transformation provided the lowest %-errors and the best fits. Moreover, the type of the calibration model significantly affected not only the fit of the data but also the matrix effects when evaluating them based on the comparison of calibration curve slopes. Indeed, based on the used calibration model, the matrix effects calculated from calibration slopes ranged from +92% to – 72% for α-tocopherol and from −77% to +19% in the case of δ-tocotrienol. Thus, it was crucial to calculate the matrix effect by Matuszewski's post-extraction approach at six concentration levels. Indeed, a strong concentration dependence was observed for all optimized sample preparation methods, even if the stable isotopically labelled internal standards (SIL-IS) were used for compensation. The significant differences between individual concentration levels and compounds were observed, even when the tested calibration range covered only one order of magnitude. In methods with wider calibration ranges, the inappropriate use of calibration slope comparison instead of the post-extraction addition approach could result in false negative results of matrix effects. In the selected example of vitamin E, solid-phase extraction was the least affected by matrix effects when used in interferent removal mode, but supported liquid extraction resulted in the highest recoveries. We showed that the calibration model, the use of a SIL-IS, and the analyte concentration level played a crucial role in the matrix effects. Moreover, the matrix effects can significantly differ for compounds with similar physicochemical properties and close retention times. Thus, in all bioanalytical applications, where different analytes are typically determined in one analytical run, it is necessary to carefully select the data processing in addition to the method for the sample preparation, SIL-IS, and chromatography.

Development and validation of prediction models for fetal growth restriction and birthweight: an individual participant data meta-analysis.

Fetal growth restriction is associated with perinatal morbidity and mortality. Early identification of women having at-risk fetuses can reduce perinatal adverse outcomes. To assess the predictive performance of existing models predicting fetal growth restriction and birthweight, and if needed, to develop and validate new multivariable models using individual participant data. Individual participant data meta-analyses of cohorts in International Prediction of Pregnancy Complications network, decision curve analysis and health economics analysis. Pregnant women at booking. External validation of existing models (9 cohorts, 441,415 pregnancies); International Prediction of Pregnancy Complications model development and validation (4 cohorts, 237,228 pregnancies). Maternal clinical characteristics, biochemical and ultrasound markers. fetal growth restriction defined as birthweight <10th centile adjusted for gestational age and with stillbirth, neonatal death or delivery before 32 weeks' gestation birthweight. First, we externally validated existing models using individual participant data meta-analysis. If needed, we developed and validated new International Prediction of Pregnancy Complications models using random-intercept regression models with backward elimination for variable selection and undertook internal-external cross-validation. We estimated the study-specific performance (c-statistic, calibration slope, calibration-in-the-large) for each model and pooled using random-effects meta-analysis. Heterogeneity was quantified using τ2 and 95% prediction intervals. We assessed the clinical utility of the fetal growth restriction model using decision curve analysis, and health economics analysis based on National Institute for Health and Care Excellence 2008 model. Of the 119 published models, one birthweight model (Poon) could be validated. None reported fetal growth restriction using our definition. Across all cohorts, the Poon model had good summary calibration slope of 0.93 (95% confidence interval 0.90 to 0.96) with slight overfitting, and underpredicted birthweight by 90.4 g on average (95% confidence interval 37.9 g to 142.9 g). The newly developed International Prediction of Pregnancy Complications-fetal growth restriction model included maternal age, height, parity, smoking status, ethnicity, and any history of hypertension, pre-eclampsia, previous stillbirth or small for gestational age baby and gestational age at delivery. This allowed predictions conditional on a range of assumed gestational ages at delivery. The pooled apparent c-statistic and calibration were 0.96 (95% confidence interval 0.51 to 1.0), and 0.95 (95% confidence interval 0.67 to 1.23), respectively. The model showed positive net benefit for predicted probability thresholds between 1% and 90%. In addition to the predictors in the International Prediction of Pregnancy Complications-fetal growth restriction model, the International Prediction of Pregnancy Complications-birthweight model included maternal weight, history of diabetes and mode of conception. Average calibration slope across cohorts in the internal-external cross-validation was 1.00 (95% confidence interval 0.78 to 1.23) with no evidence of overfitting. Birthweight was underestimated by 9.7 g on average (95% confidence interval -154.3 g to 173.8 g). We could not externally validate most of the published models due to variations in the definitions of outcomes. Internal-external cross-validation of our International Prediction of Pregnancy Complications-fetal growth restriction model was limited by the paucity of events in the included cohorts. The economic evaluation using the published National Institute for Health and Care Excellence 2008 model may not reflect current practice, and full economic evaluation was not possible due to paucity of data. International Prediction of Pregnancy Complications models' performance needs to be assessed in routine practice, and their impact on decision-making and clinical outcomes needs evaluation. The International Prediction of Pregnancy Complications-fetal growth restriction and International Prediction of Pregnancy Complications-birthweight models accurately predict fetal growth restriction and birthweight for various assumed gestational ages at delivery. These can be used to stratify the risk status at booking, plan monitoring and management. This study is registered as PROSPERO CRD42019135045. This award was funded by the National Institute for Health and Care Research (NIHR) Health Technology Assessment programme (NIHR award ref: 17/148/07) and is published in full in Health Technology Assessment; Vol. 28, No. 14. See the NIHR Funding and Awards website for further award information.