Brier Score Research Articles

External validation of the modified CTP score based on ammonia to predict survival in patients with cirrhosis after TIPS placement

This study aimed to perform the first external validation of the modified Child-Turcotte-Pugh score based on plasma ammonia (aCTP) and compare it with other risk scoring systems to predict survival in patients with cirrhosis after transjugular intrahepatic portosystemic shunt (TIPS) placement. We retrospectively reviewed 473 patients from three cohorts between January 2016 and June 2022 and compared the aCTP score with the Child-Turcotte-Pugh (CTP) score, albumin-bilirubin (ALBI), model for end-stage liver disease (MELD) and sodium MELD (MELD-Na) in predicting transplant-free survival by the concordance index (C-index), area under the receiver operating characteristic curve, calibration plot, and decision curve analysis (DCA) curve. The median follow-up time was 29 months, during which a total of 62 (20.74%) patients died or underwent liver transplantation. The survival curves for the three aCTP grades differed significantly. Patients with aCTP grade C had a shorter expected lifespan than patients with aCTP grades A and B (P < 0.0001). The aCTP score showed the best discriminative performance using the C-index compared with other scores at each time point during follow-up, it also showed better calibration in the calibration plot and the lowest Brier scores, and it also showed a higher net benefit than the other scores in the DCA curve. The aCTP score outperformed the other risk scores in predicting survival after TIPS placement in patients with cirrhosis and may be useful for risk stratification and survival prediction.

Machine learning predictions of the adverse events of different treatments in patients with ischemic left ventricular systolic dysfunction.

This study aimed to develop several new machine learning models based on hibernating myocardium to predict the major adverse cardiac events(MACE) of ischemic left ventricular systolic dysfunction(LVSD) patients receiving either percutaneous coronary intervention(PCI) or optimal medical therapy(OMT). This study included 329 LVSD patients, who were randomly assigned to the training or validation cohort. Least absolute shrinkage and selection operator(LASSO) regression was used to identify variables associated with MACE. Subsequently, various machine learning models were established. Model performance was compared using receiver operating characteristic(ROC) curves, the Brier score(BS), and the concordance index(C-index). A total of 329 LVSD patients were retrospectively enrolled between January 2016 and December 2021. Utilizing LASSO regression analysis, five factors were selected. Based on these factors, RSF, GBM, XGBoost, Cox, and DeepSurv models were constructed. In the development and validation cohorts, the C-indices were 0.888 vs. 0.955 (RSF). The RSF model (0.991 vs. 0.982 vs. 0.980) had the highest area under the ROC curve (AUC) compared with the other models. The BS (0.077 vs. 0.095vs. 0.077) of RSF model were less than 0.25 at 12, 18, and 24months. This study developed a novel predictive model based on RSF to predict MACE in LVSD patients who underwent either PCI or OMT.

Low Stroke Volume Predicts Deterioration in Intermediate-Risk Pulmonary Embolism: Prospective Study.

Prognosis and management of patients with intermediate-risk pulmonary embolism (PE) is challenging. We investigated whether stroke volume may be used to identify the subset of this population at increased risk of clinical deterioration or PE-related death. Our secondary objective was to compare echocardiographic measurements of patients who received escalated interventions vs anticoagulation monotherapy. We selected patients with intermediate-risk PE, who had comprehensive echocardiography within 18 hours of PE diagnosis and before any escalated interventions, from a PE registry populated by 11 emergency departments. Echocardiographers measured right ventricle (RV) size, tricuspid annular plane systolic excursion (TAPSE), and stroke volume (SV) using velocity time integral (VTI) by left ventricular (LV) outflow tract Doppler or two-dimensional method of discs (MOD). The primary outcome was a composite of PE-related death, cardiac arrest, catecholamine administration for sustained hypotension, or emergency respiratory intervention during the index hospitalization. Secondary outcome was escalated intervention with reperfusion or extracorporeal membrane oxygenation therapy. Of 370 intermediate-risk PE patients (mean age 64.0 ± 15.5years, 38.1% male), 39 (10.5%) had the primary outcome. These 39 patients had lower mean SV regardless of measurement method than those without the primary outcome: SV MOD 36.2 vs 49.9 milliliters (mL), P < 0.001; SV Doppler 41.7 vs 57.2mL, P = 0.003; VTI 13.6 vs 17.9 centimeters [cm], P = 0.003. Patients with primary outcome also had lower mean TAPSE than those without (1.54 vs 1.81cm, P = 0.003). Multivariable models, selecting SV as predictor, had area under the receiver operating curve of 0.8 and Brier score 0.08. The best echocardiographic predictor of our primary outcome was SV MOD (odds ratio 0.72 [0.53, 0.94], P = 0.02). Patients who received escalated interventions had significantly lower SV or surrogate measurements, greater RV dilatation, and lower RV systolic function than patients who received anticoagulation monotherapy. Low stroke volume was a predictor of clinical deterioration and PE-related death. Low SV may be used to identify a subset of intermediate-risk PE patients, who are higher risk (intermediate-high risk), and for whom escalated interventions should be considered.

Interpretability-based machine learning for predicting the risk of death from pulmonary inflammation in Chinese intensive care unit patients.

The objective of this research was to create a machine learning predictive model that could be easily interpreted in order to precisely determine the risk of premature death in patients receiving intensive care after pulmonary inflammation. In this study, information from the China intensive care units (ICU) Open Source database was used to examine data from 2790 patients who had infections between January 2019 and December 2020. A 7:3 ratio was used to randomly assign the whole patient population to training and validation groups. This study used six machine learning techniques: logistic regression, random forest, gradient boosting tree, extreme gradient boosting tree (XGBoost), multilayer perceptron, and K-nearest neighbor. A cross-validation grid search method was used to search the parameters in each model. Eight metrics were used to assess the models' performance: accuracy, precision, recall, F1 score, area under the curve (AUC) value, Brier score, Jordon's index, and calibration slope. The machine methods were ranked based on how well they performed in each of these metrics. The best-performing models were selected for interpretation using both the Shapley Additive exPlanations (SHAP) and Local interpretable model-agnostic explanations (LIME) interpretable techniques. A subset of the study cohort's patients (120/1668, or 7.19%) died in the hospital following screening for inclusion and exclusion criteria. Using a cross-validated grid search to evaluate the six machine learning techniques, XGBoost showed good discriminative ability, achieving an accuracy score of 0.889 (0.874-0.904), precision score of 0.871 (0.849-0.893), recall score of 0.913 (0.890-0.936), F1 score of 0.891 (0.876-0.906), and AUC of 0.956 (0.939-0.973). Additionally, XGBoost exhibited excellent performance with a Brier score of 0.050, Jordon index of 0.947, and calibration slope of 1.074. It was also possible to create an interactive internet page using the XGBoost model. By identifying patients at higher risk of early mortality, machine learning-based mortality risk prediction models have the potential to significantly improve patient care by directing clinical decision making and enabling early detection of survival and mortality issues in patients with pulmonary inflammation disease.

Development and Validation of Prognostic Models Using Radiomic Features from Pre-Treatment Positron Emission Tomography (PET) Images in Head and Neck Squamous Cell Carcinoma (HNSCC) Patients.

High-dimensional radiomics features derived from pre-treatment positron emission tomography (PET) images offer prognostic insights for patients with head and neck squamous cell carcinoma (HNSCC). Using 124 PET radiomics features and clinical variables (age, sex, stage of cancer, site of cancer) from a cohort of 232 patients, we evaluated four survival models-penalized Cox model, random forest, gradient boosted model and support vector machine-to predict all-cause mortality (ACM), locoregional recurrence/residual disease (LR) and distant metastasis (DM) probability during 36, 24 and 24 months of follow-up, respectively. We developed models with five-fold cross-validation, selected the best-performing model for each outcome based on the concordance index (C-statistic) and the integrated Brier score (IBS) and validated them in an independent cohort of 102 patients. The penalized Cox model demonstrated better performance for ACM (C-statistic = 0.70, IBS = 0.12) and DM (C-statistic = 0.70, IBS = 0.08) while the random forest model displayed better performance for LR (C-statistic = 0.76, IBS = 0.07). We conclude that the ML-based prognostic model can aid clinicians in quantifying prognosis and determining effective treatment strategies, thereby improving favorable outcomes in HNSCC patients.

Class imbalance on medical image classification: towards better evaluation practices for discrimination and calibration performance.

This work aims to assess standard evaluation practices used by the research community for evaluating medical imaging classifiers, with a specific focus on the implications of class imbalance. The analysis is performed on chest X-rays as a case study and encompasses a comprehensive model performance definition, considering both discriminative capabilities and model calibration. We conduct a concise literature review to examine prevailing scientific practices used when evaluating X-ray classifiers. Then, we perform a systematic experiment on two major chest X-ray datasets to showcase a didactic example of the behavior of several performance metrics under different class ratios and highlight how widely adopted metrics can conceal performance in the minority class. Our literature study confirms that: (1) even when dealing with highly imbalanced datasets, the community tends to use metrics that are dominated by the majority class; and (2) it is still uncommon to include calibration studies for chest X-ray classifiers, albeit its importance in the context of healthcare. Moreover, our systematic experiments confirm that current evaluation practices may not reflect model performance in real clinical scenarios and suggest complementary metrics to better reflect the performance of the system in such scenarios. Our analysis underscores the need for enhanced evaluation practices, particularly in the context of class-imbalanced chest X-ray classifiers. We recommend the inclusion of complementary metrics such as the area under the precision-recall curve (AUC-PR), adjusted AUC-PR, and balanced Brier score, to offer a more accurate depiction of system performance in real clinical scenarios, considering metrics that reflect both, discrimination and calibration performance. This study underscores the critical need for refined evaluation metrics in medical imaging classifiers, emphasizing that prevalent metrics may mask poor performance in minority classes, potentially impacting clinical diagnoses and healthcare outcomes. Common scientific practices in papers dealing with X-ray computer-assisted diagnosis (CAD) systems may be misleading. We highlight limitations in reporting of evaluation metrics for X-ray CAD systems in highly imbalanced scenarios. We propose adopting alternative metrics based on experimental evaluation on large-scale datasets.

Deep learning models for predicting the survival of patients with hepatocellular carcinoma based on a surveillance, epidemiology, and end results (SEER) database analysis

Hepatocellular carcinoma (HCC) is a common malignancy with poor survival and requires long-term follow-up. Hence, we collected information on patients with Primary Hepatocellular Carcinoma in the United States from the Surveillance, Epidemiology, and EndResults (SEER) database. We used this information to establish a deep learning with a multilayer neural network (the NMTLR model) for predicting the survival rate of patients with Primary Hepatocellular Carcinoma. HCC patients pathologically diagnosed between January 2011 and December 2015 in the SEER (Surveillance, Epidemiology, and End Results) database of the National Cancer Institute of the United States were selected as study subjects. We utilized two deep learning-based algorithms (DeepSurv and Neural Multi-Task Logistic Regression [NMTLR]) and a machine learning-based algorithm (Random Survival Forest [RSF]) for model training. A multivariable Cox Proportional Hazards (CoxPH) model was also constructed for comparison. The dataset was randomly divided into a training set and a test set in a 7:3 ratio. The training dataset underwent hyperparameter tuning through 1000 iterations of random search and fivefold cross-validation. Model performance was assessed using the concordance index (C-index), Brier score, and Integrated Brier Score (IBS). The accuracy of predicting 1-year, 3-year, and 5-year survival rates was evaluated using Receiver Operating Characteristic (ROC) curves, calibration plots, and Area Under the Curve (AUC). The primary outcomes were the 1-year, 3-year, and 5-year overall survival rates. Models were developed using DeepSurv, NMTLR, RSF, and Cox Proportional Hazards regression. Model differentiation was evaluated using the C-index, calibration with concordance plots, and risk stratification capability with the log-rank test. The study included 2197 HCC patients, randomly divided into a training cohort (70%, n = 1537) and a testing cohort (30%, n = 660). Clinical characteristics between the two cohorts showed no significant statistical difference (p > 0.05). The deep learning models outperformed both RSF and CoxPH models, with C-indices of 0.735 (NMTLR) and 0.731 (DeepSurv) in the test dataset. The NMTLR model demonstrated enhanced accuracy and well-calibrated survival estimates, achieving an Area Under the Curve (AUC) of 0.824 for 1-year survival predictions, 0.813 for 3-year, and 0.803 for 5-year survival rates. This model's superior calibration and discriminative ability enhance its utility for clinical prognostication in Primary Hepatocellular Carcinoma. We deployed the NMTLR model as a web application for clinical practice. The NMTLR model have potential advantages over traditional linear models in prognostic assessment and treatment recommendations. This novel analytical approach may provide reliable information on individual survival and treatment recommendations for patients with primary liver cancer.

International Multi-Specialty Expert Physician Preoperative Identification of Extranodal Extension n Oropharyngeal Cancer Patients using Computed Tomography: Prospective Blinded Human Inter-Observer Performance Evaluation.

Extranodal extension (pENE) is a critical prognostic factor in oropharyngeal cancer (OPC) that drives therapeutic disposition. Determination of pENE from radiological imaging has been associated with high inter-observer variability. However, the impact of clinician specialty on human observer performance of imaging-detected extranodal extension (iENE) remains poorly understood. To characterize the impact of clinician specialty on the accuracy of pre-operative iENE in human papillomavirus-positive (HPV+) OPC using computed tomography (CT) images. This prospective observational human performance study analyzed pre-therapy CT images from 24 HPV+ OPC patients, with duplication of 6 scans (n=30) of which 21 were pathologically confirmed pENE. Thirty-four expert observers, including 11 radiologists, 12 surgeons, and 11 radiation oncologists, independently assessed these scans for iENE and reported human-detected radiologic criteria and observer confidence. The primary outcomes included accuracy, sensitivity, specificity, area under the receiver operating characteristic curve (AUC), and Brier score for each physician, compared to ground-truth pENE. The significance of radiographic signs for prediction of pENE were determined through logistic regression analysis. Fleiss' kappa measured interobserver agreement, and Hanley-MacNeil AUC discrimination testing. Median accuracy across all specialties was 0.57 (95%CI 0.39 to 0.73), with no specialty showing discriminate performance greater than random estimation (median AUC 0.64, 95%CI 0.44 to 0.83). Significant differences between radiologists and surgeons in Brier scores (0.33 vs. 0.26, p < 0.01), radiation oncologists and surgeons in sensitivity (0.48 vs. 0.69, p > 0.1), and radiation oncologists and radiologists/surgeons in specificity (0.89 vs. 0.56, p > 0.1). Indistinct capsular contour and nodal necrosis were significant predictors of correct pENE status among all specialties. Interobserver agreement was weak for all the radiographic criteria, regardless of specialty (κ<0.6). Multiobserver testing shows physician discrimination of HPV+OPC pENE on pre-operative CT remains non-different than blind guessing, with high interrater variability and low diagnostic accuracy, regardless of clinician specialty. While minor differences in diagnostic performance among specialties are noted, they do not significantly affect the overall poor agreement and discrimination rates observed. The findings underscore the need for further research into automated detection systems or enhanced imaging techniques to improve the accuracy and reliability of iENE assessments in clinical practice.

A novel higher performance nomogram based on explainable machine learning for predicting mortality risk in stroke patients within 30 days based on clinical features on the first day ICU admission

BackgroundThis study aimed to develop a higher performance nomogram based on explainable machine learning methods, and to predict the risk of death of stroke patients within 30 days based on clinical characteristics on the first day of intensive care units (ICU) admission.MethodsData relating to stroke patients were extracted from the Medical Information Marketplace of the Intensive Care (MIMIC) IV and III database. The LightGBM machine learning approach together with Shapely additive explanations (termed as explain machine learning, EML) was used to select clinical features and define cut-off points for the selected features. These selected features and cut-off points were then evaluated using the Cox proportional hazards regression model and Kaplan-Meier survival curves. Finally, logistic regression-based nomograms for predicting 30-day mortality of stroke patients were constructed using original variables and variables dichotomized by cut-off points, respectively. The performance of two nomograms were evaluated in overall and individual dimension.ResultsA total of 2982 stroke patients and 64 clinical features were included, and the 30-day mortality rate was 23.6% in the MIMIC-IV datasets. 10 variables (“sofa (sepsis-related organ failure assessment)”, “minimum glucose”, “maximum sodium”, “age”, “mean spo2 (blood oxygen saturation)”, “maximum temperature”, “maximum heart rate”, “minimum bun (blood urea nitrogen)”, “minimum wbc (white blood cells)” and “charlson comorbidity index”) and respective cut-off points were defined from the EML. In the Cox proportional hazards regression model (Cox regression) and Kaplan-Meier survival curves, after grouping stroke patients according to the cut-off point of each variable, patients belonging to the high-risk subgroup were associated with higher 30-day mortality than those in the low-risk subgroup. The evaluation of nomograms found that the EML-based nomogram not only outperformed the conventional nomogram in NIR (net reclassification index), brier score and clinical net benefits in overall dimension, but also significant improved in individual dimension especially for low “maximum temperature” patients.ConclusionsThe 10 selected first-day ICU admission clinical features require greater attention for stroke patients. And the nomogram based on explainable machine learning will have greater clinical application.

Assessing calibration and bias of a deployed machine learning malnutrition prediction model within a large healthcare system

Malnutrition is a frequently underdiagnosed condition leading to increased morbidity, mortality, and healthcare costs. The Mount Sinai Health System (MSHS) deployed a machine learning model (MUST-Plus) to detect malnutrition upon hospital admission. However, in diverse patient groups, a poorly calibrated model may lead to misdiagnosis, exacerbating health care disparities. We explored the model’s calibration across different variables and methods to improve calibration. Data from adult patients admitted to five MSHS hospitals from January 1, 2021 - December 31, 2022, were analyzed. We compared MUST-Plus prediction to the registered dietitian’s formal assessment. Hierarchical calibration was assessed and compared between the recalibration sample (N = 49,562) of patients admitted between January 1, 2021 - December 31, 2022, and the hold-out sample (N = 17,278) of patients admitted between January 1, 2023 - September 30, 2023. Statistical differences in calibration metrics were tested using bootstrapping with replacement. Before recalibration, the overall model calibration intercept was −1.17 (95% CI: −1.20, −1.14), slope was 1.37 (95% CI: 1.34, 1.40), and Brier score was 0.26 (95% CI: 0.25, 0.26). Both weak and moderate measures of calibration were significantly different between White and Black patients and between male and female patients. Logistic recalibration significantly improved calibration of the model across race and gender in the hold-out sample. The original MUST-Plus model showed significant differences in calibration between White vs. Black patients. It also overestimated malnutrition in females compared to males. Logistic recalibration effectively reduced miscalibration across all patient subgroups. Continual monitoring and timely recalibration can improve model accuracy.

Development of machine learning models to predict perioperative blood transfusion in hip surgery

BackgroundAllogeneic Blood transfusion is common in hip surgery but is associated with increased morbidity. Accurate prediction of transfusion risk is necessary for minimizing blood product waste and preoperative decision-making. The study aimed to develop machine learning models for predicting perioperative blood transfusion in hip surgery and identify significant risk factors.MethodsData of patients undergoing hip surgery between January 2013 and October 2021 in the Peking Union Medical College Hospital were collected to train and test predictive models. The primary outcome was perioperative red blood cell (RBC) transfusion within 72 h of surgery. Fourteen machine learning algorithms were established to predict blood transfusion risk incorporating patient demographic characteristics, preoperative laboratory tests, and surgical information. Discrimination, calibration, and decision curve analysis were used to evaluate machine learning models. SHapley Additive exPlanations (SHAP) was performed to interpret models.ResultsIn this study, 2431 hip surgeries were included. The Ridge Classifier performed the best with an AUC = 0.85 (95% CI, 0.81 to 0.88) and a Brier score = 0.21. Patient-related risk factors included lower preoperative hemoglobin, American Society of Anesthesiologists (ASA) Physical Status > 2, anemia, lower preoperative fibrinogen, and lower preoperative albumin. Surgery-related risk factors included longer operation time, total hip arthroplasty, and autotransfusion.ConclusionsThe machine learning model developed in this study achieved high predictive performance using available variables for perioperative blood transfusion in hip surgery. The predictors identified could be helpful for risk stratification, preoperative optimization, and outcomes improvement.

A novel score for predicting falls in community-dwelling older people: a derivation and validation study

BackgroundEarly detection of patients at risk of falling is crucial. This study was designed to develop and internally validate a novel risk score to classify patients at risk of falls.MethodsA total of 334 older people from a fall clinic in a medical center were selected. Least absolute shrinkage and selection operator (LASSO) regression was used to minimize the potential concatenation of variables measured from the same patient and the overfitting of variables. A logistic regression model for 1-year fall prediction was developed for the entire dataset using newly identified relevant variables. Model performance was evaluated using the bootstrap method, which included measures of overall predictive performance, discrimination, and calibration. To streamline the assessment process, a scoring system for predicting 1-year fall risk was created.ResultsWe developed a new model for predicting 1-year falls, which included the FRQ-Q1, FRQ-Q3, and single-leg standing time (left foot). After internal validation, the model showed good discrimination (C statistic, 0.803 [95% CI 0.749–0.857]) and overall accuracy (Brier score, 0.146). Compared to another model that used the total FRQ score instead, the new model showed better continuous net reclassification improvement (NRI) [0.468 (0.314–0.622), P < 0.01], categorical NRI [0.507 (0.291–0.724), P < 0.01; cutoff: 0.200–0.800], and integrated discrimination [0.205 (0.147–0.262), P < 0.01]. The variables in the new model were subsequently incorporated into a risk score. The discriminatory ability of the scoring system was similar (C statistic, 0.809; 95% CI, 0.756–0.861; optimism-corrected C statistic, 0.808) to that of the logistic regression model at internal bootstrap validation.ConclusionsThis study resulted in the development and internal verification of a scoring system to classify 334 patients at risk for falls. The newly developed score demonstrated greater accuracy in predicting falls in elderly people than did the Timed Up and Go test and the 30-Second Chair Sit-Stand test. Additionally, the scale demonstrated superior clinical validity for identifying fall risk.

Advancing Glaucoma Diagnosis: Employing Confidence-Calibrated Label Smoothing Loss for Model Calibration

ObjectiveThe aim of our research is to enhance the calibration of machine learning models for glaucoma classification through a specialized loss function named Confidence-Calibrated Label Smoothing Loss (CC-LS). This approach is specifically designed to refine model calibration without compromising accuracy by integrating label smoothing and confidence penalty techniques, tailored to the specifics of glaucoma detection. DesignThis study focuses on the development and evaluation of a calibrated deep learning model. ParticipantsThe study employs fundus images from both external datasets-ORIGA (482 normal, 168 glaucoma) and REFUGE (720 normal, 80 glaucoma)-and an extensive internal dataset (4,639 images per category), aiming to bolster the model's generalizability. The model's clinical performance is validated using a comprehensive test set (47,913 normal, 1,629 glaucoma) from the internal dataset. MethodsThe CC-LS loss function seamlessly integrates label smoothing, which tempers extreme predictions to avoid overfitting, with confidence-based penalties. These penalties deter the model from expressing undue confidence in incorrect classifications. Our study aims at training models using the CC-LS and comparing their performance with those trained using conventional loss functions. Main Outcome MeasuresThe model's precision is evaluated using metrics like the Brier score, sensitivity, specificity, and the false positive rate, alongside qualitative heatmap analyses for a holistic accuracy assessment. ResultsPreliminary findings reveal that models employing the CC-LS mechanism exhibit superior calibration metrics, as evidenced by a Brier score of 0.098, along with notable accuracy measures: sensitivity of 81%, specificity of 80%, and weighted accuracy of 80%. Importantly, these enhancements in calibration are achieved without sacrificing classification accuracy. ConclusionsThe Confidence-Calibrated Label Smoothing Loss presents a significant advancement in the pursuit of deploying machine learning models for glaucoma diagnosis. By improving calibration, the CC-LS ensures that clinicians can interpret and trust the predictive probabilities, making AI-driven diagnostic tools more clinically viable. From a clinical standpoint, this heightened trust and interpretability can potentially lead to more timely and appropriate interventions, thereby optimizing patient outcomes and safety.

Prediction of CCL2 in breast cancer based on enhanced MRI radiomics model with integrated bioinformatics analysis and machine learning.

e12509 Background: Breast cancer is the most common malignant tumor in women. According to the 2023 Cancer Statistics Report, breast cancer has surpassed lung cancer to become the most common malignant tumor. Its incidence and mortality both rank first among female malignant tumors. This study proposes non-invasive prediction of CCL2 mRNA expression in breast cancer tissue through enhanced MRI radiomics, while integrating bioinformatics analysis to explore the underlying molecular mechanisms and their association with the immune microenvironment. Methods: Excluding criteria were applied to select primary solid tumor samples with RNA-seq data (n = 840) from the TCGA-BRCA database. Enhanced MRI imaging data (n = 98) intersecting with the TCGA-BRCA data were obtained from TCIA-BRCA. The imaging data were randomly divided into training (n = 70) and testing sets (n = 28) in a 7:3 ratio. A feature subset was selected, and a model was constructed using the support vector machine (SVM) algorithm in the training set. The predictive performance of the model was evaluated using receiver operating characteristic (ROC) curves, calibration curves, and the Hosmer-Lemeshow goodness-of-fit test. The calibration of the radiomics prediction model was assessed using the Brier score, and the clinical utility of the radiomics prediction model was demonstrated by drawing decision curves (DCA). External validation of the model was performed using a validation set. The radiomics score (RS) was calculated using the LR radiomics model and categorized as a binary variable (Low/High RS). Subsequently, KEGG pathway, immune-related genes, and gene mutation analysis were conducted based on the CCL2 high and low expression levels. Results: A total of 1810 radiomic features were obtained with ICC values ≥ 0.75. Subsequently, the mRMR (Maximum relevance, minimum redundancy) method, and the optimal 7 feature subsets were further identified using the RFE (Recursive feature elimination) algorithm. The SVM model demonstrated favorable predictive performance, with an AUC value of 0.822 for the training set and 0.779 for the validation set, as indicated by the ROC curve. Calibration curve and Hosmer-Lemeshow goodness-of-fit test revealed good consistency between the radiomic predictive model for the probability of high gene expression and the actual values (P &gt; 0.05). Additionally, the decision curve analysis (DCA) demonstrated high clinical utility of the model. The TNF-α signaling pathway was found to influence the occurrence and development of breast cancer. Furthermore, the degree of Plasma Cell infiltration was observed to be highest in breast cancer. Conclusions: The CCL2 model constructed using a combination of machine learning and radiomic features, can non-invasively predict the prognosis of breast cancer treatment and exhibits good generalizability.

Impacts of Initial Condition Perturbation Blending in 10- and 40-Member Convection-Allowing Ensemble Forecasts

Abstract A series of convection-allowing 36-h ensemble forecasts during the 2018 spring season are used to better understand the impacts of ensemble configuration and blending different sources of initial condition (IC) perturbation. Ten- and forty-member ensemble configurations are initialized with the multiscale IC perturbations generated as a product of convective-scale data assimilation (MULTI) and initialized with the MULTI IC perturbations blended with IC perturbations downscaled from coarser-resolution ensembles (BLEND). The forecast performance of both precipitation and nonprecipitation variables is consistently improved by the larger ensemble size. The benefit of the larger ensemble is largely, but not entirely, due to compensating for underdispersion in the fixed-physics ensemble configuration. A consistent improvement in precipitation forecast skill results from blending in the 10-member ensemble configuration, corresponding to a reduction in the ensemble calibration error (i.e., reliability component of Brier score). In the 40-member ensemble configuration, the advantage of blending is limited to the ∼18–22-h lead times at all precipitation thresholds and the ∼35–36-h lead times at the lowest threshold, both corresponding to an improved resolution component of the Brier score. The advantage of blending in the 40-member ensemble during the diurnal convection maximum of ∼18–22-h lead times is primarily due to cases with relatively weak synoptic-scale forcing, while advantages at later lead times beyond ∼30-h lead time are most prominent on cases with relatively strong synoptic-scale forcing. The impacts of blending and ensemble configuration on forecasts of nonprecipitation variables are generally consistent with the impacts on the precipitation forecasts.

Explainable machine learning predicts survival of retroperitoneal liposarcoma: A study based on the SEER database and external validation in China

AbstractObjectiveWe have developed explainable machine learning models to predict the overall survival (OS) of retroperitoneal liposarcoma (RLPS) patients. This approach aims to enhance the explainability and transparency of our modeling results.MethodsWe collected clinicopathological information of RLPS patients from The Surveillance, Epidemiology, and End Results (SEER) database and allocated them into training and validation sets with a 7:3 ratio. Simultaneously, we obtained an external validation cohort from The First Affiliated Hospital of Naval Medical University (Shanghai, China). We performed LASSO regression and multivariate Cox proportional hazards analysis to identify relevant risk factors, which were then combined to develop six machine learning (ML) models: Cox proportional hazards model (Coxph), random survival forest (RSF), ranger, gradient boosting with component‐wise linear models (GBM), decision trees, and boosting trees. The predictive performance of these ML models was evaluated using the concordance index (C‐index), the integrated cumulative/dynamic area under the curve (AUC), and the integrated Brier score, as well as the Cox–Snell residual plot. We also used time‐dependent variable importance, analysis of partial dependence survival plots, and the generation of aggregated survival SHapley Additive exPlanations (SurvSHAP) plots to provide a global explanation of the optimal model. Additionally, SurvSHAP (t) and survival local interpretable model‐agnostic explanations (SurvLIME) plots were used to provide a local explanation of the optimal model.ResultsThe final ML models are consisted of six factors: patient's age, gender, marital status, surgical history, as well as tumor's histopathological classification, histological grade, and SEER stage. Our prognostic model exhibits significant discriminative ability, particularly with the ranger model performing optimally. In the training set, validation set, and external validation set, the AUC for 1, 3, and 5 year OS are all above 0.83, and the integrated Brier scores are consistently below 0.15. The explainability analysis of the ranger model also indicates that histological grade, histopathological classification, and age are the most influential factors in predicting OS.ConclusionsThe ranger ML prognostic model exhibits optimal performance and can be utilized to predict the OS of RLPS patients, offering valuable and crucial references for clinical physicians to make informed decisions in advance.

The development of a prediction model based on random survival forest for the prognosis of non- Hodgkin lymphoma: A prospective cohort study in China

Background and objectiveThe pathological staging of non-Hodgkin lymphoma (NHL) is complex, the clinical manifestations are varied, and the prognosis differ considerably. To provide a useful reference for early detection and effective treatment of NHL, we developed a random survival forest (RSF) prognostic model based on machine learning (ML) algorithms using prospective cohort data collected from Chongqing Cancer Hospital from Jan 1, 2017 to Dec 31, 2019 (n = 1449) to compare with the traditional cornerstone method Cox proportional hazards (CPH) model and evaluate the predictability of the model. MethodsPatients were randomly split into a training cohort (TC) and validation cohort (VC) based on 65/35 ratio. The least absolute shrinkage and selection operator (LASSO) regression analysis was used to extracted the important features. And the RSF was modeled to explore the prognostic factors impacting the overall survival (OS) of patients with NHLs in the TC and validated in the VC. The C-index, the Integrated Brier Score (IBS), Kaplan-Meir method, the receiver operating characteristic (ROC) curve, and the area under the ROC curve (AUC) were selected to measure performances and discriminations of the models. In addition, individual survival probability predicted for NHL patients. ResultsAccording to the features extracted by LASSO model and univariable Cox model, 16 variables were selected to develop the RSF model with log-rank splitting rule, which were age, ethnicity, medical insurance, Ann Arbor stage, pathology, targeted-therapy, chemo-therapy, peripheral blood neutrophil count to lymphocyte count ratio (NLR), peripheral blood platelet count to lymphocyte count ratio (PLR), serum lactate dehydrogenase (LDH), CD4/CD8, platelet (PLT), absolute neutrophil count (ANC), lymphocyte (LYM), B-symptoms, and (CPR) were important prognostic factors. Compared to the CPH model (C-index = 0.748, IBS = 0.166), the RSF model (C-index = 0.786, IBS = 0.165) is outperformed in predictability and accuracy. The AUC of the RSF model to estimate the 1-, 3-, and 5-year OS in TC were 0.847, 0.847, and 0.809, respectively; while those in the CPH were 0.816, 0.803, and 0.750, respectively. ConclusionsTo provide practical implications for the implementation of individualized therapy, the study constructed a high-performed RSF model and reveal that it outperformed the traditional model CPH. And the RSF model ranked the risk variables. In addition, we stratified the risk of NHL patients and estimated individual survival probability based on the RSF model.

Developing a Machine Learning Model to Predict 180-day Readmission for Elderly Patients with Angina.

Readmission of elderly angina patients has become a serious problem, with a dearth of available prediction tools for readmission assessment. The objective of this study was to develop a machine learning (ML) model that can predict 180-day all-cause readmission for elderly angina patients. The clinical data for elderly angina patients was retrospectively collected. Five ML algorithms were used to develop prediction models. Area under the receiver operating characteristic curve (AUROC), area under the precision recall curve (AUPRC), and the Brier score were applied to assess predictive performance. Analysis by Shapley additive explanations (SHAP) was performed to evaluate the contribution of each variable. A total of 1502 elderly angina patients (45.74% female) were enrolled in the study. The extreme gradient boosting (XGB) model showed good predictive performance for 180-day readmission (AUROC = 0.89; AUPRC = 0.91; Brier score = 0.21). SHAP analysis revealed that the number of medications, hematocrit, and chronic obstructive pulmonary disease were important variables associated with 180-day readmission. An ML model can accurately identify elderly angina patients with a high risk of 180-day readmission. The model used to identify individual risk factors can also serve to remind clinicians of appropriate interventions that may help to prevent the readmission of patients.

Machine learning for prediction of chronic kidney disease progression: Validation of the Klinrisk model in the CANVAS Program and CREDENCE trial.

To validate the Klinrisk machine learning model for prediction of chronic kidney disease (CKD) progression in patients with type 2 diabetes in the pooled CANVAS/CREDENCE trials. We externally validated the Klinrisk model for prediction of CKD progression, defined as 40% or higher decline in estimated glomerular filtration rate (eGFR) or kidney failure. Model performance was assessed for prediction up to 3 years with the area under the receiver operating characteristic curve (AUC), Brier scores and calibration plots of observed and predicted risks. We compared performance of the model with standard of care using eGFR (G1-G4) and urine albumin-creatinine ratio (A1-A3) Kidney Disease Improving Global Outcomes (KDIGO) heatmap categories. The Klinrisk model achieved an AUC of 0.81 (95% confidence interval [CI] 0.78-0.83) at 1 year, and 0.88 (95% CI 0.86-0.89) at 3 years. The Brier scores were 0.020 (0.018-0.022) and 0.056 (0.052-0.059) at 1 and 3 years, respectively. Compared with the KDIGO heatmap, the Klinrisk model had improved performance at every interval (P < .01). The Klinrisk machine learning model, using routinely collected laboratory data, was highly accurate in its prediction of CKD progression in the CANVAS/CREDENCE trials. Integration of the model in electronic medical records or laboratory information systems can facilitate risk-based care.

Deep Survival Models Can Improve Long-Term Mortality Risk Estimates from Chest Radiographs

Deep learning has recently demonstrated the ability to predict long-term patient risk and its stratification when trained on imaging data such as chest radiographs. However, existing methods formulate estimating patient risk as a binary classification, typically ignoring or limiting the use of temporal information, and not accounting for the loss of patient follow-up, which reduces the fidelity of estimation and limits the prediction to a certain time horizon. In this paper, we demonstrate that deep survival and time-to-event prediction models can outperform binary classifiers at predicting mortality and risk of adverse health events. In our study, deep survival models were trained to predict risk scores from chest radiographs and patient demographic information in the Prostate, Lung, Colorectal, and Ovarian (PLCO) cancer screening trial (25,433 patient data points used in this paper) for 2-, 5-, and 10-year time horizons. Binary classification models that predict mortality at these time horizons were built as baselines. Compared to the considered alternative, deep survival models improve the Brier score (5-year: 0.0455 [95% CI, 0.0427–0.0482] vs. 0.0555 [95% CI, (0.0535–0.0575)], p &lt; 0.05) and expected calibration error (ECE) (5-year: 0.0110 [95% CI, 0.0080–0.0141] vs. 0.0747 [95% CI, 0.0718–0.0776], p &lt; 0.05) for those fixed time horizons and are able to generate predictions for any time horizon, without the need to retrain the models. Our study suggests that deep survival analysis tools can outperform binary classification in terms of both discriminative performance and calibration, offering a potentially plausible solution for forecasting risk in clinical practice.