External Validation Study Research Articles

Diagnostic performance of deep learning for infectious keratitis: a systematic review and meta-analysis

Infectious keratitis (IK) is the leading cause of corneal blindness globally. Deep learning (DL) is an emerging tool for medical diagnosis, though its value in IK is unclear. We aimed to assess the diagnostic accuracy of DL for IK and its comparative accuracy with ophthalmologists. In this systematic review and meta-analysis, we searched EMBASE, MEDLINE, and clinical registries for studies related to DL for IK published between 1974 and July 16, 2024. We performed meta-analyses using bivariate models to estimate summary sensitivities and specificities. This systematic review was registered with PROSPERO (CRD42022348596). Of 963 studies identified, 35 studies (136,401 corneal images from >56,011 patients) were included. Most studies had low risk of bias (68.6%) and low applicability concern (91.4%) in all domains of QUADAS-2, except the index test domain. Against the reference standard of expert consensus and/or microbiological results (seven external validation studies; 10,675 images), the summary estimates (95% CI) for sensitivity and specificity of DL for IK were 86.2% (71.6-93.9) and 96.3% (91.5-98.5). From 28 internal validation studies (16,059 images), summary estimates for sensitivity and specificity were 91.6% (86.8-94.8) and 90.7% (84.8-94.5). Based on seven studies (4007 images), DL and ophthalmologists had comparable summary sensitivity [89.2% (82.2-93.6) versus 82.2% (71.5-89.5); P=0.20] and specificity [(93.2% (85.5-97.0) versus 89.6% (78.8-95.2); P=0.45]. DL models may have good diagnostic accuracy for IK and comparable performance to ophthalmologists. These findings should be interpreted with caution due to the image-based analysis that did not account for potential correlation within individuals, relatively homogeneous population studies, lack of pre-specification of DL thresholds, and limited external validation. Future studies should improve their reporting, data diversity, external validation, transparency, and explainability to increase the reliability and generalisability of DL models for clinical deployment. NIH, Wellcome Trust, MRC, Fight for Sight, BHP, and ESCRS.

Assessment of Early Vascular Aging Ambulatory Score (EVAAs): A Large Population-based External Validation Study.

Pulse Wave Velocity (PWV) remains the gold-standard method to assess Early Vascular Aging (EVA) defined by arterial stiffness. However, its high cost, time-consuming process, and need for qualified medical staff shows the importance of identifying alternative methods for the EVA evaluation. In order to simplify the process of assessing patients' EVA, we recently developed the Early Vascular Aging Ambulatory score (EVAAs), a simple tool to predict the risk of EVA. The aim of the present study was the external validation of EVAAs in an independent population. Eight hundred seventy-nine (46.3% men) patients who were referred to our Hypertension ESH Excellence Center were included in this study. The mean age was 46.43 ± 22.87 years. EVA was evaluated in two different ways. The first assessment included c-f PWV values, whereas the second one included EVAAs without the direct measurement of carotid-femoral PWV. The null hypothesis was that the prediction of EVA based on EVAAs does not present any statistically significant difference compared to the prediction based on the calculation from c-f PWV. Mean squared error (MSE) was used for the assessment of the null hypothesis, which was found to be 0.40. The results revealed that the EVAAs show the probability of EVA with 0.98 sensitivity and 0.75 specificity. The EVAAs present 95% positive predictive value and 92% negative predictive value. Our study revealed that EVAAs could be as reliable as the carotid-femoral PWV to identify patients with EVA. Hence, we hope that EVAAs will be a useful tool in clinical practice.

Risk assessment using DEVI and CARPREG -1 score to predict maternal adverse cardiac cvents in pregnancies with valvular heart disease: external validation study from MPAC registry

Abstract Background Even though many risk assessment tools in pregnant women with valvular heart disease (VHD) are available, validation studies are few that assess their performance in diverse settings, which is needed prior to the application of routine clinical practice. Objectives To validate and establish the clinical utility of two risk stratification tools – DEVI (VHD-specific tool) and CARPREG-I in predicting adverse cardiac events in pregnant patients with VHD. Methods This cohort study involved consecutive pregnancies complicated with VHD enrolled in the prospective MPAC registry from July 2016 to December 2019(1). The ability to discriminate those with and without adverse cardiac outcomes was assessed using the area under the curve for both the DEVI and CARPREG-1 scores. Performance was assessed through discrimination and calibration characteristics. Clinical utility was evaluated with Decision Curve Analysis. Results Among 1029 pregnancies complicated with heart disease in the MPAC registry cohort, 604(52.3%) with valvular heart disease were included in this analysis. One or more composite adverse cardiac events occurred in 70 (11.6%) pregnancies during antenatal or the early postpartum period. Mitral regurgitation was the most common lesion (67.7%). In assessing the pregnancy outcomes, those who continued past 20 weeks were included, with the majority (92.0%) delivered at term, and the cesarean rates were 33.6% (n=203). The most common adverse event was heart failure (n=32, 5.3%), most of which (68.6%) occurred in the postnatal period. There were 11 maternal deaths (1.8%) during the study period, six among those with mechanical heart valves (four prosthetic valve thrombosis), two among those with severe mitral stenosis and pulmonary hypertension and two with severe mitral regurgitation and infective endocarditis. The area under the receiver operating characteristic curve (AUC) was 0.766, with 95% confidence intervals (CI) 0.703- 0.828 for DEVI and 0.703 (95%CI 0.644- 0.763) for CARPREG-I models. Calibration plots suggested that the DEVI score overestimates risk at higher probabilities, whereas the CARPREG-I score underestimates the risk at most probabilities. Decision curve analysis demonstrated that both models were useful across predicted probability thresholds between 10% and 60%. Conclusion In this external validation study in pregnant women with primary rheumatic VHD, DEVI and CARPREG-I scores showed good discrimination ability and clinical utility across various probabilities. Both can be used to classify those who develop adverse cardiac outcomes during pregnancy and childbirth. However, both models need recalibration to improve the agreement between the predicted and observed events prior to routine application in diverse clinical settings.Figure 1.a and b ROC and Decision Curves

Validation of Fetal Medicine Foundation charts for fetal growth in twins: nationwide Danish cohort study.

To assess the validity of the Fetal Medicine Foundation (FMF) chorionicity-specific models for fetal growth in twin pregnancy. This was an external validation study of the FMF models using a nationwide Danish cohort of twin pregnancies. The cohort included all dichorionic (DC) and monochorionic diamniotic (MCDA) twin pregnancies with an estimated delivery date between 2008 and 2018, which satisfied the following inclusion criteria: two live fetuses at the first-trimester ultrasound scan (11-14 weeks' gestation); biometric measurements available for the calculation of estimated fetal weight (EFW) using the Hadlock-3 formula; and delivery of two liveborn infants. Validation involved assessing the distributional properties of the models and estimating the mean EFW Z-score deviations. Additionally, the models were applied to pregnancies that delivered preterm and attended non-scheduled visits (complicated pregnancies). Overall, 8542 DC and 1675 MCDA twin pregnancies met the inclusion criteria. In DC twins, 17 084 fetuses were evaluated at a total of 95 346 ultrasound scans, of which 44.5% were performed at scheduled visits in pregnancies carried to 37 + 0 weeks or later. The median number of growth scans per DC twin fetus from 20 + 0 weeks onwards was four. The model showed good agreement with the validation cohort for scheduled visits in DC twins delivered at 37 + 0 weeks or later (mean EFW Z-score, -0.14 ± 1.05). In MCDA twins, 3350 fetuses underwent 31 632 eligible ultrasound scans, of which 59.5% were performed at scheduled visits in pregnancies carried to 36 + 0 weeks or later. The median number of growth scans per MCDA twin fetus from 16 + 0 weeks onwards was 10. The model showed favorable agreement with the validation cohort for scheduled visits in MCDA twins delivered at 36 + 0 weeks or later (mean EFW Z-score, -0.09 ± 1.01). Non-scheduled visits and preterm delivery before 37 + 0 weeks for DC twins and before 36 + 0 weeks for MCDA twins corresponded with smaller weight estimates, which was consistent with the study's definition of complicated pregnancy. The FMF models provide a good fit for EFW measurements in our Danish national cohort of uncomplicated twin pregnancies assessed at routine scans. Therefore, the FMF models establish robust criteria for subsequent investigations and potential clinical applications. Future research should focus on exploring the consequences of clinical implementation, particularly regarding the identification of twins that are small-for-gestational age, as they are especially susceptible to adverse perinatal outcome. © 2024 The Author(s). Ultrasound in Obstetrics & Gynecology published by John Wiley & Sons Ltd on behalf of International Society of Ultrasound in Obstetrics and Gynecology.

Public evidence on AI products for digital pathology

Novel products applying artificial intelligence (AI)-based methods to digital pathology images are touted to have many uses and benefits. However, publicly available information for products can be variable, with few sources of independent evidence. This review aimed to identify public evidence for AI-based products for digital pathology. Key features of products on the European Economic Area/Great Britain (EEA/GB) markets were examined, including their regulatory approval, intended use, and published validation studies. There were 26 AI-based products that met the inclusion criteria and, of these, 24 had received regulatory approval via the self-certification route as General in vitro diagnostic (IVD) medical devices. Only 10 of the products (38%) had peer-reviewed internal validation studies and 11 products (42%) had peer-reviewed external validation studies. To support transparency an online register was developed using identified public evidence (https://osf.io/gb84r/), which we anticipate will provide an accessible resource on novel devices and support decision making.

Caries Detection and Classification in Photographs Using an Artificial Intelligence-Based Model-An External Validation Study.

This ex vivo diagnostic study aimed to externally validate a freely accessible AI-based model for caries detection, classification, localisation and segmentation using an independent image dataset. It was hypothesised that there would be no difference in diagnostic performance compared to previously published internal validation data. For the independent dataset, 718 dental images representing different stages of carious (n = 535) and noncarious teeth (n = 183) were retrieved from the internet. All photographs were evaluated by the dental team (reference standard) and the AI-based model (test method). Diagnostic performance was statistically determined using cross-tabulations to calculate accuracy (ACC), sensitivity (SE), specificity (SP) and area under the curve (AUC). An overall ACC of 92.0% was achieved for caries detection, with an ACC of 85.5-95.6%, SE of 42.9-93.3%, SP of 82.1-99.4% and AUC of 0.702-0.909 for the classification of caries. Furthermore, 97.0% of the cases were accurately localised. Fully and partially correct segmentation was achieved in 52.9% and 44.1% of the cases, respectively. The validated AI-based model showed promising diagnostic performance in detecting and classifying caries using an independent image dataset. Future studies are needed to investigate the validity, reliability and practicability of AI-based models using dental photographs from different image sources and/or patient groups.

Recommendations for the creation of benchmark datasets for reproducible artificial intelligence in radiology

Various healthcare domains have witnessed successful preliminary implementation of artificial intelligence (AI) solutions, including radiology, though limited generalizability hinders their widespread adoption. Currently, most research groups and industry have limited access to the data needed for external validation studies. The creation and accessibility of benchmark datasets to validate such solutions represents a critical step towards generalizability, for which an array of aspects ranging from preprocessing to regulatory issues and biostatistical principles come into play. In this article, the authors provide recommendations for the creation of benchmark datasets in radiology, explain current limitations in this realm, and explore potential new approaches.Clinical relevance statementBenchmark datasets, facilitating validation of AI software performance can contribute to the adoption of AI in clinical practice.Key PointsBenchmark datasets are essential for the validation of AI software performance.Factors like image quality and representativeness of cases should be considered.Benchmark datasets can help adoption by increasing the trustworthiness and robustness of AI.Graphical

Research progress of cardiovascular disease risk prediction models among patients with chronic kidney disease

Patients with chronic kidney disease (CKD) have a relatively high risk of cardiovascular disease (CVD). Risk stratification guided by CVD risk prediction models is essential for managing CKD populations. We reviewed the outcome events, predictive variables, modeling methods, and predictive performance of CVD risk prediction models in CKD populations. We found a large variability in predictive outcomes, number of predictors, and sample sizes across studies. The models tended to overestimate the CVD risk of CKD populations. There are few independently validated or constructed CVD risk prediction models for CKD populations in developing countries, and in particular, there is a lack of independent external validation studies of model calibration. Future studies should comply with the reporting standards of risk prediction models to better support the application of CVD risk prediction models for CKD populations.

Evaluating Combinations of Biological and Clinicopathologic Factors Linked to Poor Outcomes in Resected Colorectal Liver Metastasis: An External Validation Study.

Recent studies have suggested that certain combinations of KRAS or BRAF biomarkers with clinical factors are associated with poor outcomes and may indicate that surgery could be "biologically" futile in otherwise technically resectable colorectal liver metastasis (CRLM). However, these combinations have yet to be validated through external studies. We conducted a systematic search to identify these studies. The overall survival (OS) of patients with these combinations was evaluated in a cohort of patients treated at 11 tertiary centers. Additionally, the study investigated whether using high-risk KRAS point mutations in these combinations could be associated with particularly poor outcomes. The recommendations of four studies were validated in 1661 patients. The first three studies utilized KRAS, and their validation showed the following median and 5-year OS: (1) 30 months and 16.9%, (2) 24.3 months and 21.6%, and (3) 46.8 months and 44.4%, respectively. When analyzing only patients with high-risk KRAS mutations, median and 5-year OS decreased to: (1) 26.2 months and 0%, (2) 22.3 months and 15.1%, and (3) not reached and 44.9%, respectively. The fourth study utilized BRAF, and its validation showed a median OS of 10.4 months, with no survivors beyond 21 months. The combinations of biomarkers and clinical factors proposed to render surgery for CRLM futile, as presented in studies 1 (KRAS high-risk mutations) and 4, appear justified. In these studies, there were no long-term survivors, and survival was similar to that of historic cohorts with similar mutational profiles that received systemic therapies alone for unresectable disease.

Multivariable prediction of functional outcome after first-episode psychosis: a crossover validation approach in EUFEST and PSYSCAN.

Several multivariate prognostic models have been published to predict outcomes in patients with first episode psychosis (FEP), but it remains unclear whether those predictions generalize to independent populations. Using a subset of demographic and clinical baseline predictors, we aimed to develop and externally validate different models predicting functional outcome after a FEP in the context of a schizophrenia-spectrum disorder (FES), based on a previously published cross-validation and machine learning pipeline. A crossover validation approach was adopted in two large, international cohorts (EUFEST, n = 338, and the PSYSCAN FES cohort, n = 226). Scores on the Global Assessment of Functioning scale (GAF) at 12 month follow-up were dichotomized to differentiate between poor (GAF current < 65) and good outcome (GAF current ≥ 65). Pooled non-linear support vector machine (SVM) classifiers trained on the separate cohorts identified patients with a poor outcome with cross-validated balanced accuracies (BAC) of 65-66%, but BAC dropped substantially when the models were applied to patients from a different FES cohort (BAC = 50-56%). A leave-site-out analysis on the merged sample yielded better performance (BAC = 72%), highlighting the effect of combining data from different study designs to overcome calibration issues and improve model transportability. In conclusion, our results indicate that validation of prediction models in an independent sample is essential in assessing the true value of the model. Future external validation studies, as well as attempts to harmonize data collection across studies, are recommended.

Prediction of pre-eclampsia with machine learning approaches: Leveraging important information from routinely collected data

BackgroundGlobally, pre-eclampsia (PE) is a leading cause of maternal and perinatal morbidity and mortality. PE prediction using routinely collected data has the advantage of being widely applicable, particularly in low-resource settings. Early intervention for high-risk women might reduce PE incidence and related complications. We aimed to replicate our machine learning (ML) published work predicting another maternal condition (gestational diabetes) to (1) predict PE using routine health data, (2) identify the optimal ML model, and (3) compare it with logistic regression approach. MethodsData were from a large health service network with 48,250 singleton pregnancies between January 2016 and June 2021. Supervised ML models were employed. Maternal clinical and medical characteristics were the feature variables (predictors), and a 70/30 data split was used for training and testing the model. Predictive performance was assessed using area under the curve (AUC) and calibration plots. Shapley value analysis assessed the contribution of feature variables. ResultsThe random forest approach provided excellent discrimination with an AUC of 0.84 (95% CI: 0.82–0.86) and highest prediction accuracy (0.79); however, the calibration curve (slope of 1.21, 95% CI 1.13–1.30) was acceptable only for a threshold of 0.3 or less. The next best approach was extreme gradient boosting, which provided an AUC of 0.77 (95% CI: 0.76–0.79) and well-calibrated (slope of 0.93, 95% CI 0.85–1.01). Logistic regression provided good discrimination performance with an AUC of 0.75 (95% CI: 0.74–0.76) and perfect calibration. Nulliparous, pre-pregnancy body mass index, previous pregnancy with prior PE, maternal age, family history of hypertension, and pre-existing hypertension and diabetes were the top-ranked features in Shapley value analysis. ConclusionTwo ML models created the highest-performing prediction using routinely collected data to identify women at high risk of PE, with acceptable discrimination. However, to confirm this result and also examine model generalisability, external validation studies are needed in other settings, utilising standardised prognostic factors.

Diagnostic performance of TILs-US score and LPBC in biopsy specimens for predicting pathological complete response in patients with breast cancer.

Tumor-infiltrating lymphocytes-ultrasonography (TILs-US) score is used to predict lymphocyte-predominant breast cancer (LPBC) in surgical specimens. We aimed to compare diagnostic performance of TILs-US score for predicting pathological complete response (pCR) with that of LPBC in biopsy specimens. TILs ≥ 50% in biopsy specimens was defined as biopsy-LPBC, and TILs-US score ≥ 4 was categorized as TILs-US score-high. Basic nomogram for pCR was developed using stepwise logistic regression based on the smallest Akaike Information Criterion, and biopsy-LPBC and TILs-US score nomograms were developed by integrating biopsy-LPBC or TILs-US scores into a basic nomogram. The diagnostic performance of the nomograms for pCR was compared using area under the curve (AUC), categorical net reclassification improvement (NRI), and integrated discrimination improvement (IDI). This retrospective study evaluated 118 patients with breast cancer, including 33 (28.0%) with biopsy-LPBC, 52 (44.1%) with TILs-US score-high, with 34 (28.8%) achieving pCR. The sensitivity, specificity, positive likelihood ratio, negative likelihood ratio, and AUC for predicting pCR were 0.53, 0.82, 2.96, 0.57, and 0.68, respectively, for biopsy-LPBC, and 0.76, 0.69, 2.47, 0.34, and 0.73, respectively, for TILs-US score. The biopsy-LPBC nomogram showed significant improvements in categorical NRI (p = 0.023) and IDI (p = 0.007) but not in AUC (p = 0.25), compared with the basic nomogram. The TILs-US nomogram exhibited significant improvements in AUC (p = 0.039), categorical NRI (p = 0.010), and IDI (p < 0.001). The TILs-US score may serve as a novel marker for prediction of pCR in patients with breast cancer. An external validation study is warranted to confirm our findings.

Automated detection of central retinal artery occlusion using OCT imaging via explainable deep learning

ObjectiveTo demonstrate the capability of a deep learning model to detect central retinal artery occlusion (CRAO), a retinal pathology with significant clinical urgency, using optical coherence tomography (OCT) data. DesignRetrospective, external validation study analyzing OCT and clinical baseline data of two institutions via deep learning classification analysis. SubjectsPatients presenting to the University Medical Center Tübingen and the University Medical Center Hamburg-Eppendorf in Germany. MethodsOCT-data of patients suffering from CRAO, differential diagnosis with (sub-)acute visual loss (central retinal vein occlusion, diabetic macular edema, non-arteritic ischemic optic neuropathy) and from controls were expertly graded and distinguished into the three groups.Our methodological approach involved a nested multiclass 5-fold cross-validation classification scheme. Main Outcome MeasuresArea under the curve (AUC) ResultsThe optimal performance of our algorithm was observed using 30 epochs, complemented by an early stopping mechanism to prevent overfitting. Our model followed a multiclass approach, distinguishing among the three different classes: Control CRAO and differential diagnoses. The evaluation was conducted by the “one vs. all” area under the receiver operating characteristics (ROC) curve (AUC) method. The results demonstrated AUC of 0.96 (95% confidence interval (CI): ± 0.01), 0.99 (95% CI: ± 0.00), 0.90 (95% CI: ± 0.03) for each class, respectively. ConclusionsOur machine learning algorithm (MLA) exhibited a high AUC, as well as sensitivity and specificity in detecting CRAO and the differential classes, respectively. These findings underscore the potential for deploying MLAs in the identification of less common etiologies within an acute emergency clinical setting.

External validation of the PRECE nomogram model in a Central America cohort

Introduction: This is an external validation study of the PRECE prostate cancer nomogram in a Central American population. Methods: We present 102 consecutive cases of robotic radical prostatectomy, performed with preservation of the anterior fascia and we use the PRECE nomogram as a guide to preserve the prostatic neurovascular pedicles. We compared the predicted extra-capsular extension from the PRECE nomogram to the final prostatectomy pathology. Results: Analysis of post-prostatectomy pathological samples revealed that 61% patients had pT2; 27% had a pT3a and 12% had a pT3b, respectively The ROC curve for the PRECE nomogram at one (1) mm showed a model AUC of 0.91 (95% CI), which implies a high agreement with the PRECE nomogram in the prediction of extraprostatic disease. Conclusion: This is the first report of external validation of the PRECE nomogram in a Central American population. PRECE demonstrated high discrimination for the prediction of extraprostatic disease in an independent Latin American cohort. More external validation studies, from different geographic settings, are expected to confirm the reliability and reproducibility of PRECE in other clinical settings.

Validation of machine learning-based risk stratification scores for patients with acute coronary syndrome treated with percutaneous coronary intervention

Abstract Aims This study aimed to validate the machine learning-based Global Registry of Acute Coronary Events (GRACE) 3.0 score and PRAISE (Prediction of Adverse Events following an Acute Coronary Syndrome) in patients with acute coronary syndrome (ACS) treated with percutaneous coronary intervention (PCI) for predicting mortality. Methods and results Data of consecutive patients with ACS treated with PCI in a tertiary centre in the Netherlands between 2014 and 2021 were used for external validation. The GRACE 3.0 score for predicting in-hospital mortality was evaluated in 2759 patients with non-ST-elevation acute coronary syndrome (NSTE-ACS) treated with PCI. The PRAISE score for predicting one-year mortality was evaluated in 4347 patients with ACS treated with PCI. Both risk scores were compared with the GRACE 2.0 score. The GRACE 3.0 score showed excellent discrimination [c-statistic 0.90 (95% CI 0.84, 0.94)] for predicting in-hospital mortality, with well-calibrated predictions (calibration-in-the large [CIL] −0.19 [95% CI −0.45, 0.07]). The PRAISE score demonstrated moderate discrimination [c-statistic 0.75 (95% CI 0.70, 0.80)] and overestimated the one-year risk of mortality [CIL −0.56 (95% CI −0.73, −0.39)]. Decision curve analysis demonstrated that the GRACE 3.0 score offered improved risk prediction compared with the GRACE 2.0 score, while the PRAISE score did not. Conclusion This study in ACS patients treated with PCI provides suggestive evidence that the GRACE 3.0 score effectively predicts in-hospital mortality beyond the GRACE 2.0 score. The PRAISE score demonstrated limited potential for predicting one-year mortality risk. Further external validation studies in larger cohorts including patients without PCI are warranted.

Validation of the HAS-BLED scale for the assessment of bleeding risk in patients on anticoagulation therapy with a diagnosis of venous thromboembolic disease.

Several models have been developed to assess bleeding risk in patients with venous thromboembolism, such as HAS-BLED, but their external validity has not been adequately assessed. The objective of the study was to evaluate the discriminative ability and calibration of the HAS-BLED scale for predicting 1-month bleeding risk in patient's anticoagulated for venous thromboembolism. External validation study of a prediction model based on a retrospective cohort of patients with venous thromboembolism treated between November 2019 and January 2022. Calibration of the HAS-BLED scale was evaluated using the Hosmer-Lemeshow test and the ratio of observed to expect events within each risk category. Discriminatory ability was assessed using the area under the curve (AUC) of a receiver operating characteristic curve. We included 735 patients (median age 64 years, female sex 55.2%), pulmonary embolism was diagnosed in most patients (60.7%), and 4.9% presented bleeding events. Regarding calibration, the HAS-BLED scale systematically underestimates the risk both in the general population (ROE 3.76, p < 0.001) and in cancer patients (ROE 4.16). The Hosmer-Lemeshow test rejected the hypothesis of adequate calibration (p < 0.001). Discriminatory ability was limited both in the general population (AUC = 0.57, 95% confidence interval [CI]: 0.48-0.66) and in the subgroup with active cancer (AUC = 0.53, 95% CI: 0.36-0.69). The HAS-BLED scale in patients with venous thromboembolism underestimates the risk of bleeding at 1 month and has a low ability to discriminate high-risk patients. Cautious interpretation of the scale is recommended until additional evidence is available.

Predicting nosocomial pneumonia risk in level-1 trauma patients: An external validation study using the trauma quality improvement program

BackgroundEarly identification of patients at risk of nosocomial pneumonia enables the opportunity for preventative measures, which may improve survival and reduce costs. Therefore, this study aimed to externally validate an existing prediction model (issued by Croce et al.) to predict nosocomial pneumonia in patients admitted to US level-1 trauma centers. MethodsA retrospective cohort study including patients admitted to level-1 trauma centers and registered in the TQIP, a US nationwide trauma registry, admitted between 2013–2015 and 2017–2019. The main outcome was total nosocomial pneumonia for the first period and ventilator-associated pneumonia (VAP) for the second. Model discrimination and calibration were assessed before and after recalibration. ResultsThe study comprised 902,231 trauma patients (N2013–2015 ​= ​180,601; N2017–2019 ​= ​721,630), with a median age of 52 in both periods, 64–65 ​% male, and approximately 90 ​% sustaining blunt traumatic injury. The median Injury Severity Scores were 13 (2013–2015) versus 9 (2017–2019); median Glasgow Coma Scale scores were 15. Nosocomial pneumonia incidence was 4.4 ​%, VAP incidence was 0.7 ​%. The original model demonstrated good to excellent discrimination for both periods (c-statistic2013–2015 0.84, 95%CI 0.83–0.84; c-statistic2017–2019 0.92, 95%CI 0.91–0.92). After recalibration, discriminatory capacity and calibration for the lower predicted probabilities improved. ConclusionsThe Croce model can identify patients admitted to US level-1 trauma centers at risk of total nosocomial pneumonia and VAP. Implementing (modified) Croce models in route trauma clinical practice could guide judicious use of preventative measures and prescription of additional non-invasive preventative measures (e.g., increased monitoring, pulmonary physiotherapy) to decrease the occurrence of nosocomial pneumonia in at-risk patients.

Biomarker-enhanced cardiovascular risk prediction in patients with cancer: a prospective cohort study.

Continuously improving cancer-specific survival puts a growing proportion of cancer patients at risk of major adverse cardiovascular events (MACE), but tailored tools for cardiovascular risk prediction remain unavailable. To assess a broad panel of cardiovascular biomarkers and risk factors for the prediction of MACE and cardiovascular death in cancer patients. In total, 2192 patients with newly diagnosed or recurrent cancer were followed prospectively for the occurrence of 2-year MACE and 5-year cardiovascular death. Univariable and multivariable risk models were fit to assess independent associations of cardiovascular risk factors and biomarkers with adverse outcomes, and a risk score was developed. Traditional cardiovascular risk factors and selected cancer types were linked to higher MACE risk. While levels of Lp(a), CRP, and GDF-15 did not associate with MACE, levels of ICAM-1, P-/E-/L-selectins, and NT-proBNP were independently linked to 2-year MACE risk. A clinical risk score was derived, assigning+1 point for male sex, smoking, and age of ≥60 years and+2 points for atherosclerotic disease, yielding a bootstrapped C-statistic of 0.76 (95% CI: 0.71-0.81) for the prediction of 2-year MACE. Implementation of biomarker data conferred improved performance (0.83, 95% CI: 0.78-0.88), with a simplified model showing similar performance (0.80, 95% CI: 0.74-0.86). The biomarker-enhanced and simplified prediction models achieved a C-statistic of 0.82 (95% CI: 0.71-0.93) and 0.74 (95% CI: 0.64-0.83) for the prediction of 5-year cardiovascular death. Biomarker-enhanced risk prediction strategies allow the identification of cancer patients at high risk of MACE and cardiovascular death. While external validation studies are ongoing, this first-of-its-kind risk score may provide the basis for personalized cardiovascular risk assessment across cancer entities.

External validation of the Hong Kong Chinese non-laboratory risk models and scoring algorithm for case finding of prediabetes and diabetes mellitus in primary care.

Two Hong Kong Chinese non-laboratory-based prediabetes/diabetes mellitus (pre-DM/DM) risk models were developed using logistic regression (LR) and machine learning, respectively. We aimed to evaluate the models' validity in case finding of pre-DM/DM in a Chinese primary care (PC) population. We also evaluated the validity of a risk-scoring algorithm derived from the LR model. This was a cross-sectional external validation study on Chinese adults, without a prior DM diagnosis, who were recruited from public/private PC clinics in Hong Kong. A total of 1,237 participants completed a questionnaire on the models' predictors. Of that, 919 underwent blood glucose testing. The primary outcome was the models' and the algorithm's sensitivity in finding pre-DM/DM cases. The secondary outcomes were the models' and the algorithm's specificity, positive/negative predictive values, discrimination and calibration. The models' sensitivity were 0.70 (machine learning) and 0.72 (LR). Both showed good external discrimination (area under the receiver operating characteristic curve: machine learning 0.744, LR 0.739). The risks estimated by the models were lower than the observed incidence, indicating poor calibration. Both models were more effective among participants with lower pretest probabilities; that is, age 18-44 years. The algorithm's sensitivity was 0.77 at the cut-off score of ≥16 out of 41. This study showed the validity of the models and the algorithm for finding pre-DM/DM cases in a Chinese PC population in Hong Kong. They can facilitate more cost-effective identification of high-risk individuals for blood testing to diagnose pre-DM/DM in PC. Further studies should recalibrate the models for more precise risk estimation in PC populations.

Psychosis metabolic risk calculator (PsyMetRiC) in early psychosis: External validation study in Finland.

Accurate detection of cardiometabolic risk in early psychosis is crucial to reducing somatic morbidity and mortality in people with psychotic disorders. We conducted an external validation of the psychosis metabolic risk calculator (PsyMetRiC), a cardiometabolic risk prediction tool developed in the UK and tailored for young people with psychosis. We compared the predictive accuracy and clinical usefulness of PsyMetRiC and a general population-based risk prediction tool for type 2 diabetes, the Finnish Diabetes Risk Score (FINDRISC). We included first-episode psychosis and ultra-high-risk for psychosis patients without metabolic syndrome aged 18-35 years from the Helsinki Early Psychosis and Turku Early Psychosis Study cohorts. We tested two versions of PsyMetRiC: the full model including age, sex, ethnicity, body-mass index, smoking status, prescription of metabolically-active antipsychotic medication, high-density lipoprotein, and triglyceride concentrations, and the partial-model excluding biochemical predictors, and the simplified FINDRISC including BMI, sex, systolic blood pressure, and fasting glucose. Discrimination, calibration, and decision curve analyses were used to assess the predictive performance and clinical usefulness of both PsyMetRiC and FINDRISC. We performed a site-specific re-calibration of PsyMetRiC (PsyMetRiC-Fi). The study sample consisted of 278 individuals (all White European ethnicity, 58.6% male, mean age 24.8 years, 37.8% smoking, mean BMI 23.5). Discrimination was marginally better in the PsyMetRiC full model (C = 0.72, 95% CI, 0.59-0.82) compared with partial model (C = 0.70, 95% CI 0.59-0.80) or FINDRISC (C = 0.63, 95% CI 0.54-0.71). Calibration plots displayed evidence of minor miscalibration for PsyMetRiC, which corrected following recalibration. Miscalibration was more pronounced for FINDRISC. Decision curve analysis showed that PsyMetRiC offers likely clinical usefulness in improving cardiometabolic risk management in early psychosis compared with giving everyone or no one an intervention. PsyMetRiC has utility in predicting cardiometabolic risk in Finnish patients with early psychosis. It has better discriminatory accuracy and offers more accurate risk prediction compared to other available strategies.