Poor Calibration Research Articles

IMPACT OF DNA METHYLATION PROfiLING IN REfiNING MENINGIOMA RECURRENCE RISK STRATIfiCATION AND CLINICAL FOLLOW UP

Abstract AIMS To assess clinical impact of the additive information gained from Integrated Meningioma Risk Score on the risk of meningioma recurrence. METHOD DNA methylation (Illumina EPIC Array) analysis was performed on all recurrent meningioma tumours, primary tumours CNS5 WHO grade 2, 3 and grade 1 with unusual morphological features from Dec 2021 to Jan 2024. DNA methylation analysis was performed, and data files were processed on the DKFZ Heidelberg Classifier v12.5/6 (molecularneuropathology.org). The output from the classifier including methylation class combined with WHO Grade and chromosomal loss (1p, 6q,14q) was used to produce an integrated meningioma risk score for recurrence. RESULTS 70 meningioma cases were profiled. This included 68 cranial meningiomas, ((WHO grade 1, n=19; recurrences=7), (grade 2, n=47; recurrences =9), (grade 3 n=2; recurrences=1)) and 2 spinal meningiomas (grade 1 n=1, grade2 n=1). 3 tumours were excluded due to poor calibration scores. Of the 67 cases the diagnosis was revised in 1 case following DNA methylation analysis due to presence of CDKN2A/B loss. Molecular-morphologic integrated risk score was calculated for all 67 cases. In 10% of overall patients the predicted risk of meningioma recurrence was increased to either intermediate or high, whilst 32% had the risk lowered mainly from intermediate to low. Interestingly, 1 patient who underwent a gross total resection of a WHO Grade 2 meningioma, experienced recurrence with the integrated meningioma risk score indicating low risk of recurrence, which is contrary to the clinical aggressiveness of the tumour. CONCLUSION Meningioma integrated risk stratification by DNA methylation profiling has shown concordance of integrated risk score with WHO Grade in 58% of cases. In 42% of cases the predicted risk was altered when compared to the WHO Grade alone. These results need to be further validated with long term data on patient outcomes.

Large language model uncertainty proxies: discrimination and calibration for medical diagnosis and treatment.

The inability of large language models (LLMs) to communicate uncertainty is a significant barrier to their use in medicine. Before LLMs can be integrated into patient care, the field must assess methods to estimate uncertainty in ways that are useful to physician-users. Evaluate the ability for uncertainty proxies to quantify LLM confidence when performing diagnosis and treatment selection tasks by assessing the properties of discrimination and calibration. We examined confidence elicitation (CE), token-level probability (TLP), and sample consistency (SC) proxies across GPT3.5, GPT4, Llama2, and Llama3. Uncertainty proxies were evaluated against 3 datasets of open-ended patient scenarios. SC discrimination outperformed TLP and CE methods. SC by sentence embedding achieved the highest discriminative performance (ROC AUC 0.68-0.79), yet with poor calibration. SC by GPT annotation achieved the second-best discrimination (ROC AUC 0.66-0.74) with accurate calibration. Verbalized confidence (CE) was found to consistently overestimate model confidence. SC is the most effective method for estimating LLM uncertainty of the proxies evaluated. SC by sentence embedding can effectively estimate uncertainty if the user has a set of reference cases with which to re-calibrate their results, while SC by GPT annotation is the more effective method if the user does not have reference cases and requires accurate raw calibration. Our results confirm LLMs are consistently over-confident when verbalizing their confidence (CE).

An evaluation of source-blending impact on the calibration of SKA EoR experiments

ABSTRACT Twenty-one-centimetre signals from the Epoch of Reionization (EoR) are expected to be detected in the low-frequency radio window by the next-generation interferometers, particularly the Square Kilometre Array (SKA). However, precision data analysis pipelines are required to minimize the systematics within an infinitesimal error budget. Consequently, there is a growing need to characterize the sources of errors in EoR analysis. In this study, we identify one such error origin, namely source blending, which is introduced by the overlap of objects in the densely populated observing sky under SKA1-Low’s unprecedented sensitivity and resolution, and evaluate its two-fold impact in both the spatial and frequency domains using a novel hybrid evaluation (HEVAL) pipeline combining end-to-end simulation with an analytic method to mimic EoR analysis pipelines. Sky models corrupted by source blending induce small but severe frequency-dependent calibration errors when coupled with astronomical foregrounds, impeding EoR parameter inference with strong additive residuals in the two-dimensional power spectrum space. We report that additive residuals from poor calibration against sky models with blending ratios of 5 and 0.5 per cent significantly contaminate the EoR window. In contrast, the sky model with a 0.05 per cent blending ratio leaves little residual imprint within the EoR window, therefore identifying a blending tolerance at approximately 0.05 per cent. Given that the SKA observing sky is estimated to suffer from an extended level of blending, strategies involving de-blending, frequency-dependent error mitigation, or a combination of both, are required to effectively attenuate the calibration impact of source-blending defects.

Framework to process vehicles uncertain locations for intelligent transportation

Many factors affect the precision and accuracy of location data. These factors include, but not limited to, environmental obstructions (e.g., high buildings and forests), hardware issues (e.g., malfunctioning and poor calibration), and privacy concerns (e.g., users denying consent to fine-grained location tracking). These factors lead to uncertainty about users’ location which in turn affects the quality of location-aware services. This paper proposes a novel framework called UMove, which stands for uncertain movements, to manage the trajectory of moving objects under location uncertainty. The UMove framework employs the connectivity (i.e., links between edges) and constraints (i.e., travel time and distance) on road network graphs to reduce the uncertainty of the object’s past, present, and projected locations. To accomplish this, UMove incorporates (i) a set-based pruning algorithm to reduce or eliminate uncertainty from imprecise trajectories; and (ii) a wrapper that can extend user-defined probability models designed to predict future locations of moving objects under uncertainty. Intensive experimental evaluations based on real data sets of GPS trajectories collected by Didi Chuxing in China prove the efficiency of the proposed UMove framework. In terms of accuracy, for past exact-location inference, UMove achieves rates from 88% to 97% for uncertain regions with sizes of 75 meters and 25 meters respectively; for future exact-location inference, accuracy rates reach up to 72% and 82% for 75 meters and 25 meters of uncertain regions.

Assessing a prediction model for depression risk using an early adolescent sample with self‐reported depression

AbstractBackgroundMajor depressive disorder (MDD) in adolescence is a risk factor for poor physical and psychiatric outcomes in adulthood, with earlier age of onset associated with poorer outcomes. Identifying Depression Early in Adolescence Risk Score (IDEA‐RS) is a model for predicting MDD in youth aged >15 years, but replication in younger samples (<15 years) is lacking. Here, we tested IDEA‐RS in a younger sample (9–11 years) to assess whether IDEA‐RS could be applied to earlier onset depression.MethodsWe applied IDEA‐RS predictor weights to 9854 adolescents (9–11 years) from the Adolescent Brain Cognitive Development (ABCD) Study, United States. We derived incident depression outcomes from self‐reported data at 2‐year follow‐up (11–13 years): incident MDD and increase in depression symptoms (DS). Sensitivity analyses were conducted using parent‐reported data. We assessed accuracy and calibration in predicting self‐reported incident depression and compared this to a refitted model with predictor weights derived in ABCD. Lastly, we tested associations between IDEA‐RS predictors and self‐reported incident depression.ResultsExternal replication yielded better‐than‐chance discriminative capacity for self‐reported incident depression (MDD: AUC = 61.4%, 95% CI = 53.5%–69.4%; DS: AUC = 57.9%, 95% CI = 54.6%–61.3%) but showed poor calibration with overly extreme risk estimates. Re‐estimating predictor weights improved discriminative capacity (MDD: AUC = 75.9%, 95% CI = 70.3%–81.4%; DS: AUC = 64.8%, 95% CI = 61.9%–67.7%) and calibration. IDEA‐RS predictors ‘poorest level of relationship with the primary caregiver’ (OR = 4.25, 95% CI = 1.73–10.41) and ‘high/highest levels of family conflict’ (OR = 3.36 [95% CI = 1.34–8.43] and OR = 3.76 [95% CI = 1.50–9.38], respectively) showed greatest associations with self‐reported incident MDD.ConclusionsWhile IDEA‐RS yields better‐than‐chance predictions on external replication, accuracy is improved when differences between samples, such as case‐control mix, are adjusted for. IDEA‐RS may be more suited to research settings with sufficient data for refitting. Altogether, we find that IDEA‐RS can be generalisable to early adolescents after refitting and that family dysfunction may be especially impactful for this period of development.

External validation of the Hong Kong Chinese non-laboratory risk models and scoring algorithm for case finding of prediabetes and diabetes mellitus in primary care.

Two Hong Kong Chinese non-laboratory-based prediabetes/diabetes mellitus (pre-DM/DM) risk models were developed using logistic regression (LR) and machine learning, respectively. We aimed to evaluate the models' validity in case finding of pre-DM/DM in a Chinese primary care (PC) population. We also evaluated the validity of a risk-scoring algorithm derived from the LR model. This was a cross-sectional external validation study on Chinese adults, without a prior DM diagnosis, who were recruited from public/private PC clinics in Hong Kong. A total of 1,237 participants completed a questionnaire on the models' predictors. Of that, 919 underwent blood glucose testing. The primary outcome was the models' and the algorithm's sensitivity in finding pre-DM/DM cases. The secondary outcomes were the models' and the algorithm's specificity, positive/negative predictive values, discrimination and calibration. The models' sensitivity were 0.70 (machine learning) and 0.72 (LR). Both showed good external discrimination (area under the receiver operating characteristic curve: machine learning 0.744, LR 0.739). The risks estimated by the models were lower than the observed incidence, indicating poor calibration. Both models were more effective among participants with lower pretest probabilities; that is, age 18-44 years. The algorithm's sensitivity was 0.77 at the cut-off score of ≥16 out of 41. This study showed the validity of the models and the algorithm for finding pre-DM/DM cases in a Chinese PC population in Hong Kong. They can facilitate more cost-effective identification of high-risk individuals for blood testing to diagnose pre-DM/DM in PC. Further studies should recalibrate the models for more precise risk estimation in PC populations.

Evaluating the performance of the SORG machine learning algorithm for predicting discharge disposition in lumbar surgery patients

PurposeProtracted admissions following lumbar surgeries are rising, often stemming from inefficient identification of patients requiring nonhome discharge for rehabilitation. The SORG Orthopaedic Research Group at Harvard Medical School have developed a machine learning algorithm for predicting discharge following lumbar surgery. This study assessed its predictive performance on an independent tertiary centre patient cohort. MethodsMedical records were retrospectively reviewed for all elective adult lumbar disc degeneration or herniation surgeries performed between July 2017–2021 at a tertiary neurosurgical centre in the United Kingdom. Preoperative variables were collated and discharge destinations noted. Algorithm predictions were analysed using the concordance (c) statistic, Brier score and calibration plot. Positive and negative predictive values (PPV, NPV) were calculated, and a decision curve analysis (DCA) plotted. Results251 subjects were included (48.2 % female, mean age 55.3 years). 2.8 % underwent nonhome discharge. Most had surgery at 1/2 spinal levels (98.4 %) and were functionally independent (84.5 %). Algorithm predictions yielded a 0.88 c-statistic and 0.029 Brier score. The algorithm was miscalibrated to the data (calibration plot slope 1.31 and intercept -1.12). At a 0.25 threshold for nonroutine discharge risk, the PPV was 0.19 and NPV 0.98. DCA revealed limited clinical utility. ConclusionsAlgorithm predictive performance was mixed for this cohort, displaying strong discrimination but poor calibration and overestimation of nonroutine discharges. Differences in patient management practices and the low nonhome discharge rate may explain this. Larger validation studies across different healthcare systems, alongside geographically specific algorithm development, will improve predictive accuracy prior to clinical application.

An unmanned ground vehicle phenotyping-based method to generate three-dimensional multispectral point clouds for deciphering spatial heterogeneity in plant traits

Fusing three-dimensional (3D) and multispectral (MS) imaging data holds promise for high-throughput and comprehensive plant phenotyping to decipher genome-to-phenome knowledge. Acquiring high-quality 3D MS point clouds (3DMPCs) of plants remains challenging because of poor 3D data quality and limited radiometric calibration methods for plants with a complex canopy structure. Here, we present a novel 3D spatial–spectral data fusion approach to collect high-quality 3DMPCs of plants by integrating the next-best-view planning for adaptive data acquisition and neural reference field (NeREF) for radiometric calibration. This approach was used to acquire 3DMPCs of perilla, tomato, and rapeseed plants with diverse plant architecture and leaf morphological features evaluated by the accuracy of chlorophyll content and equivalent water thickness (EWT) estimation. The results showed that the completeness of plant point clouds collected by this approach was improved by an average of 23.6% compared with the fixed viewpoints alone. The NeREF-based radiometric calibration with the hemispherical reference outperformed the conventional calibration method by reducing the root mean square error (RMSE) of 58.93% for extracted reflectance spectra. The RMSE for chlorophyll content and EWT predictions decreased by 21.25% and 14.13% using partial least squares regression with the generated 3DMPCs. Collectively, our study provides an effective and efficient way to collect high-quality 3DMPCs of plants under natural light conditions, which improves the accuracy and comprehensiveness of phenotyping plant morphological and physiological traits, and thus will facilitate plant biology and genetic studies as well as crop breeding.

Systematic Evaluation of Uncertainty Calibration in Pretrained Object Detectors

AbstractIn the field of deep learning based computer vision, the development of deep object detection has led to unique paradigms (e.g., two-stage or set-based) and architectures (e.g., Faster-RCNN or DETR) which enable outstanding performance on challenging benchmark datasets. Despite this, the trained object detectors typically do not reliably assess uncertainty regarding their own knowledge, and the quality of their probabilistic predictions is usually poor. As these are often used to make subsequent decisions, such inaccurate probabilistic predictions must be avoided. In this work, we investigate the uncertainty calibration properties of different pretrained object detection architectures in a multi-class setting. We propose a framework to ensure a fair, unbiased, and repeatable evaluation and conduct detailed analyses assessing the calibration under distributional changes (e.g., distributional shift and application to out-of-distribution data). Furthermore, by investigating the influence of different detector paradigms, post-processing steps, and suitable choices of metrics, we deliver novel insights into why poor detector calibration emerges. Based on these insights, we are able to improve the calibration of a detector by simply finetuning its last layer.

External Validation of the Charlson Comorbidity Index-based Model for Survival Prediction in Thai Patients Diagnosed with Dementia

BackgroundThe Charlson Comorbidity Index (CCI) is commonly employed for predicting mortality. Nonetheless, its performance has rarely been evaluated in patients with dementia. This study aimed to examine the predictive capability of the CCI-based model for survival prediction in Thai patients diagnosed with dementia.MethodsAn external validation study was conducted using retrospective data from adults with dementia who had visited the outpatient departments at Maharaj Nakorn Chiang Mai Hospital between 2006 and 2012. The data obtained from electronic medical records included age, gender, date of dementia diagnosis and death, types of dementia, and comorbidities at the time of dementia diagnosis. The discriminative ability and calibration of the CCI-based model were estimated using Harrell’s C Discrimination Index and visualized with calibration plot. As the initial performance did not meet satisfaction, model updating and recalibration were performed.ResultsOf 702 patients, 56.9% were female. The mean age at dementia diagnosis was 75.22 (SD 9.75) year-old. During external validation, Harrell's C-statistic of the CCI-based model was 0.58 (95% CI, 0.54–0.61). The model showed poor external calibration. Model updating was subsequently performed. All updated models demonstrated a modest increase in Harrell's C-statistic. Temporal recalibration did not significantly improve the calibration of any of the updated models.ConclusionThe CCI-based model exhibited fair discriminative ability and poor calibration for predicting survival in Thai patients diagnosed with dementia. Despite attempts at model updating, significant improvements were not achieved. Therefore, it is important to consider the incorporation of other influential prognostic factors.

Assessment of Natural Reference Materials for U‐Pb Geochronology of Grossular‐Andradite Garnet

Garnet U‐Pb dating by laser ablation‐inductively coupled plasma‐mass spectrometry requires the development of matrix‐matched reference materials of variable chemistry and U mass fraction for accurate analysis. Additional calibration of existing primary reference materials is also justified based on the relatively poor calibration of some of the widely available primary reference materials that are currently utilised by the geoscience community. We present a micro sampling workflow combined with a refined ID‐TIMS methodology for the generation of high precision (~ 0.1%) U‐Pb dates from domains within garnet single crystals. Using this workflow, we calibrated two new natural andradite reference materials, the Jumbo andradite (And99; 110.34 ± 0.03 (0.04) [0.13] Ma, n = 7, MSWD = 1.21) and the Tiptop andradite (And87; 209.57 ± 0.11 (0.13) [0.26] Ma, n = 6, MSWD = 1.39). We also present additional calibration of the widely utilised Willsboro‐Lewis andradite primary reference material (And90; 1024.7 ± 9.5 (9.6) [9.6] Ma (2s; overdispersed), n = 6). Wafers of the Jumbo and Tiptop andradite reference materials are available from the authors upon request.

The IMPACT Score: Does Sex Matter?

ObjectiveThe Index for Mortality Prediction After Cardiac Transplantation (IMPACT) score is a quantitative risk index that predicts 1-yr mortality risk, derived from United Network for Organ Sharing (UNOS) data in which women are underrepresented. The validity of the IMPACT score in 1-yr mortality risk after OHT in women is unknown. The objective of this study is to assess differences in score performance by sex. We hypothesize that the IMPACT score is a poor predictor of 1-yr mortality risk after OHT in women. DesignExternal validation study including patients 18 years and older first-time single organ OHT recipients from The International Society of Heart and Lung Transplantation (ISHLT) data from 2009-2018. Demographic and clinical characteristics were compared by sex. IMPACT score was calculated and regression models constructed for the entire sample, and stratified by sex. Model discrimination was assessed with the area under the receiver operating characteristic curve (AUROC) and calibration was assessed graphically. SettingN/A InterventionsN/A ParticipantsPatients 18 years and older first time single OHT recipients from ISHLT registry from 2009-2018. Measurements and Main ResultsFor 1-yr mortality, the AUROC (95% CI) for the full sample was 0.59 (0.57-0.60); 0.58 (0.55-0.61) for women; 0.59 (0.58-0.61) for men. 1-yr mortality was 9.4% in the overall cohort, with no difference in mortality by sex (9.0% vs 9.6% women vs men, p=0.22). ConclusionsThe IMPACT score exhibited poor discrimination and calibration in the ISHLT 2009-2019 cohort, overall and by sex. There was no difference in 1-yr mortality between women and men.

Prediction of 30‐day mortality after surgery for infective endocarditis using risk scores: Insights from a European multicenter comparative validation study

BackgroundIt remains unclear today whether risk scores created specifically to predict early mortality after cardiac operations for infective endocarditis (IE) outperform or not the European System for Cardiac Operative Risk Evaluation II (EuroSCORE II). MethodsPerioperative data and outcomes from a European multicenter series of patients undergoing surgery for definite IE were retrospectively reviewed. Only the cases with known pathogen and without missing values for all considered variables were retained for analyses. A comparative validation of EuroSCORE II and five specific risk scores for early mortality after surgery for IE – (1) STS-IE (Society of Thoracic Surgeons for IE); (2) PALSUSE (Prosthetic valve, Age ≥70, Large intra-cardiac destruction, Staphylococcus spp, Urgent surgery, Sex (female), EuroSCORE ≥10); (3) ANCLA (Anemia, New York Heart Association class IV, Critical state, Large intra-cardiac destruction, surgery on thoracic Aorta); (4) AEPEI II (Association pour l’Étude et la Prévention de l'Endocardite Infectieuse II); (5) APORTEI (Análisis de los factores PROnósticos en el Tratamiento quirúrgico de la Endocarditis Infecciosa) – was carried out using calibration plot and receiver-operating characteristic curve analysis. Areas under the curve (AUCs) were compared 1:1 according to the Hanley-McNeil's method. The agreement between APORTEI score and EuroSCORE II of the 30-day mortality prediction after surgery was also appraised. ResultsA total of 1,012 patients from five European university-affiliated centers underwent 1,036 cardiac operations, with a 30-day mortality after surgery of 9.7%. All IE-specific risk scores considered achieved better results than EuroSCORE II in terms of calibration; AEPEI II and APORTEI score showed the best performances. Despite poor calibration, EuroSCORE II overcame in discrimination every specific risk score (AUC, 0.751 vs. 0.693 or less, p=0.01 or less). For a higher/lesser than 20% expected mortality, the agreement of prediction between APORTEI score and EuroSCORE II was 86%. ConclusionEuroSCORE II discrimination for 30-day mortality after surgery for IE was higher than five established IE-specific risk scores. AEPEI II and APORTEI score showed the best results in terms of calibration.

Risk prediction of advanced colorectal neoplasia varies by race and neighbourhood socioeconomic status

ObjectiveNeighbourhood deprivation increases the risk of colorectal neoplasia and contributes to racial disparities observed in this disease. Developing race-specific advanced colorectal neoplasia (ACN) prediction models that include neighbourhood socioeconomic status...

Clinical complexity of an Italian cardiovascular intensive care unit: the role of mortality and severity risk scores.

The identification of patients at greater mortality risk of death at admission into an intensive cardiovascular care unit (ICCU) has relevant consequences for clinical decision-making. We described patient characteristics at admission into an ICCU by predicted mortality risk assessed with noncardiac intensive care unit (ICU) and evaluated their performance in predicting patient outcomes. A total of 202 consecutive patients (130 men, 75 ± 12 years) were admitted into our tertiary-care ICCU in a 20-week period. We evaluated, on the first 24 h data, in-hospital mortality risk according to Acute Physiology and Chronic Health Evaluation II (APACHE II) and Simplified Acute Physiology Score 3 (SAPS 3); Sepsis related Organ Failure Assessment (SOFA) Score and the Mayo Cardiac intensive care unit Admission Risk Score (M-CARS) were also calculated. Predicted mortality was significantly lower than observed (5% during ICCU and 7% at discharge) for APACHE II and SAPS 3 (17% for both scores). Mortality risk was associated with older age, more frequent comorbidities, severe clinical presentation and complications. The APACHE II, SAPS 3, SOFA and M-CARS had good discriminative ability in distinguishing deaths and survivors with poor calibration of risk scores predicting mortality. In a recent contemporary cohort of patients admitted into the ICCU for a variety of acute and critical cardiovascular conditions, scoring systems used in general ICU had good discrimination for patients' clinical severity and mortality. Available scores preserve powerful discrimination but the overestimation of mortality suggests the importance of specific tailored scores to improve risk assessment of patients admitted into ICCUs.

Calibration method for binocular vision system with large field of view based on small target image splicing

In vision measurement, camera calibration has a significant impact on measurement precision. The classical target-based calibration methods require the target to occupy more than one-third of the field of view. A small-size target that does not meet the requirements results in poor calibration accuracy, while an appropriate large-size target is difficult to manufacture and inconvenient to operate. In view of the above problem, we propose a flexible and accurate calibration method based on small target image splicing to calibrate the binocular vision system with a large field of view. The spliced images and virtual large targets are constructed to extend the target size, providing better flexibility for calibration. Moreover, an optimization objective function integrating two constraints in the imaging plane and measurement space is presented to improve the calibration accuracy during the parameter optimization process. The simulation experiments and actual experiments are carried out to test the performance of the proposed method. The results demonstrate that the calibration accuracy of the proposed method using a small target is equivalent to that of Zhang’s method using a large target. Additionally, when using a same-size target, the parameter error of the proposed method is less than that of Zhang’s method, and the proposed method reduces the distance measurement error from 1.169 mm to 0.208 mm compared to Zhang’s method.

Performance of risk scores in predicting major bleeding in left ventricular assist device recipients: a comparative external validation

BackgroundImplantation of a left ventricular assist device (LVAD) is a crucial therapeutic option for selected end-stage heart failure patients. However, major bleeding (MB) complications postimplantation are a significant concern. ObjectivesWe evaluated current risk scores’ predictive accuracy for MB in LVAD recipients. MethodsWe conducted an observational, single-center study of LVAD recipients (HeartWare or HeartMate-3, November 2010-December 2022) in the Netherlands. The primary outcome was the first post-LVAD MB (according to the International Society on Thrombosis and Haemostasis [ISTH] and Interagency Registry for Mechanically Assisted Circulatory Support [INTERMACS], and INTERMACS combined with intracranial bleeding [INTERMACS+] criteria). Mortality prior to MB was considered a competing event. Discrimination (C-statistic) and calibration were evaluated for the Hypertension, Abnormal Renal/Liver Function, Stroke, Bleeding History or Predisposition, Labile INR, Elderly, Drugs/Alcohol Concomitantly score, Hepatic or Renal Disease, Ethanol Abuse, Malignancy, Older Age, Reduced Platelet Count or Function, Re-Bleeding, Hypertension, Anemia, Genetic Factors, Excessive Fall Risk and Stroke score, Anticoagulation and Risk Factors in Atrial Fibrillation score, Outpatient Bleeding Risk Index, venous thromboembolism score, atrial fibrillation score, and Utah Bleeding Risk Score (UBRS). ResultsOne hundred four patients were included (median age, 64 years; female, 20.2%; HeartWare, 90.4%; HeartMate-3, 9.6%). The cumulative MB incidence was 75.7% (95% CI 65.5%-85.9%) by ISTH and INTERMACS+ criteria and 67.0% (95% CI 56.0%-78.0%) per INTERMACS criteria over a median event-free follow-up time of 1916 days (range, 59-4521). All scores had poor discriminative ability on their intended prediction timeframe. Cumulative area under the receiving operator characteristic curve ranged from 0.49 (95% CI 0.35-0.63, venous thromboembolism-BLEED) to 0.56 (95% CI 0.47-0.65, UBRS) according to ISTH and INTERMACS+ criteria and from 0.48 (95% CI 0.40-0.56, Anticoagulation and Risk Factors in Atrial Fibrillation) to 0.56 (95% CI 0.47-0.65, UBRS) per INTERMACS criteria. All models showed poor calibration, largely underestimating MB risk. ConclusionCurrent bleeding risk scores exhibit inadequate predictive accuracy for LVAD recipients. There is a need for an accurate risk score to identify LVAD patients at high risk of MB who may benefit from patient-tailored antithrombotic therapy.

External Validation of the fullPIERS Risk Prediction Model in a U.S. Cohort of Women With Preeclampsia [ID 2683408

INTRODUCTION: Hypertensive disorders of pregnancy are a leading cause of maternal morbidity and adverse pregnancy outcomes. The fullPIERS model was developed to predict severe adverse maternal outcomes within 48 hours of admission in patients with preeclampsia. This model has not been validated in the U.S. cohort. We sought to examine whether the fullPIERS model was predictive in the U.S. cohort. METHODS: This was a retrospective study of patients with preeclampsia who delivered at 23 weeks or greater from January 1, 2010, to December 31, 2020. Our primary outcome was a composite of maternal mortality or other serious complications of preeclampsia that occurred within 48 hours of admission. Using the prediction model that considers gestational age at admission, chest pain or dyspnea, serum creatinine, platelet, aspartate transaminase, and oxygen saturation, we calculated the probability of the composite outcome. We assessed performance using the area under the curve (AUC) of the receiver operating characteristic curve. Hosmer–Lemeshow test was used to assess the goodness-of-fit. RESULTS: Of 1,552 patients with preeclampsia, 77 (5.0%) had the composite outcome within 48 hours. The fullPIERS model had an AUC of 0.73 (95% CI, 0.66–0.80). However, the Hosmer–Lemeshow test showed poor calibration with a value of P<.001. The calibration plot showed the model underestimated when the predicted probability was less than 10% and overestimated when the predicted probability was over 28%. CONCLUSION: The fullPIERS model had good AUC in the U.S. cohort of patients with preeclampsia but poor calibration. The fullPIERS model may not be clinically useful in the United States.

Relative performance evaluation of four bleeding risk scores in atrial fibrillation patients. What does the new DOAC score provide?

BackgroundRecently, the direct oral anticoagulant (DOAC) score was developed and better predicted major bleeding in DOAC-treated patients with atrial fibrillation (AF) than HASBLED did. Little is known on the new score's performance regarding other bleeding risk in AF. MethodsWe studied 14,672 patients diagnosed with AF between 2014 and 2018. During follow-up, we assessed the performance of DOAC score compared with the HASBLED, ORBIT and SWISS scores at predicting major bleeding in DOACs and non-DOACs users. Discrimination, calibration and decision curve analysis (DCA) were used to assess the risk scorer's performance. ResultsThere were 1484 (10.1%) patients on DOACs, 9730 on vitamin K antagonist (VKA), and 3458 on non-oral anticoagulants. Over a median of 3.5 years of follow-up, 79 major bleedings occurred in the DOAC patients, and 486 in the VKA patients (cumulative incidences = 7.4 and 13.9 per 100 patient-years, respectively).Amongst the DOAC patients, the DOAC score discrimination was moderate (C-statistic = 0.711), but significantly higher than HASBLED (C = 0.640; p = 0.03), ORBIT (C = 0.660; p = 0.04), and SWISS scores (C = 0.637; p = 0.002). The DCA showed higher net benefit using DOAC score compared with the remaining scores.In the VKA patients, DOAC score showed the highest discrimination (c-statistic = 0.709), followed by ORBIT (C = 0.692; p = 0.07), HASBLED and SWISS (C = 0.635 and 0.624, respectively; p < 0.01). All risk scores calibrated well, although HASBLED showed relatively poor calibration. ConclusionsThe new DOAC bleeding risk score is a valid and reasonable predictor of major bleeding over a median of 3.5 years of follow-up. Physicians can be reassured about the applicability of DOAC score for bleeding risk stratification in AF patients. RegistrationURL: https://www.clinicaltrials.gov; Unique identifier: NCT04364516

Comparative Predictive Accuracies of the Simplified Mortality Score for the Intensive Care Unit, Sepsis Severity Score, and Standard Severity Scores for 90-day Mortality in Sepsis Patients.

The standard severity scores were used for predicting hospital mortality of intensive care unit (ICU) patients. Recently, the new predictive score, Simplified Mortality Score for the ICU (SMS-ICU), was developed for predicting 90-day mortality. To validate the ability of the SMS-ICU and compare with sepsis severity score (SSS) and original severity scores for predicting 90-day mortality in sepsis patients. An analysis of retrospective data was conducted in the ICU of a university teaching hospital. Also, 90-day mortality was used for the primary outcome. A total of 1,161 patients with sepsis were included. The 90-day mortality was 42.4%. The SMS-ICU presented the area under the receiver operating characteristic curve (AUROC) of 0.71, whereas the SSS had significantly higher AUROC than that of the SMS-ICU (AUROC 0.876, p < 0.001). The acute physiology and chronic health evaluation (APACHE) II and IV, and the simplified acute physiology scores (SAPS) II demonstrated good discrimination, with an AUROC above 0.90. The SMS-ICU provides poor calibration for 90-day mortality prediction, similar to the SSS and other standard severity scores. Furthermore, 90-day mortality was underestimated by the SMS-ICU, which had a standardized mortality ratio (SMR) of 1.36. The overall performance by Brier score demonstrated that the SMS-ICU was inferior to the SSS (0.222 and 0.169, respectively). Also, SAPS II presented the best overall performance with a Brier score of 0.092. The SMS-ICU indicated lower performance compared to the SSS, standard severity scores. Consequently, modifications are required to enhance the performance of the SMS-ICU. Sathaporn N, Khwannimit B. Comparative Predictive Accuracies of the Simplified Mortality Score for the Intensive Care Unit, Sepsis Severity Score, and Standard Severity Scores for 90-day Mortality in Sepsis Patients. Indian J Crit Care Med 2024;28(4):343-348.