Calibration Metrics Research Articles

Can We Geographically Validate a Natural Language Processing Algorithm for Automated Detection of Incidental Durotomy Across Three Independent Cohorts From Two Continents?

Incidental durotomy is an intraoperative complication in spine surgery that can lead to postoperative complications, increased length of stay, and higher healthcare costs. Natural language processing (NLP) is an artificial intelligence method that assists in understanding free-text notes that may be useful in the automated surveillance of adverse events in orthopaedic surgery. A previously developed NLP algorithm is highly accurate in the detection of incidental durotomy on internal validation and external validation in an independent cohort from the same country. External validation in a cohort with linguistic differences is required to assess the transportability of the developed algorithm, referred to geographical validation. Ideally, the performance of a prediction model, the NLP algorithm, is constant across geographic regions to ensure reproducibility and model validity. Can we geographically validate an NLP algorithm for the automated detection of incidental durotomy across three independent cohorts from two continents? Patients 18 years or older undergoing a primary procedure of (thoraco)lumbar spine surgery were included. In Massachusetts, between January 2000 and June 2018, 1000 patients were included from two academic and three community medical centers. In Maryland, between July 2016 and November 2018, 1279 patients were included from one academic center, and in Australia, between January 2010 and December 2019, 944 patients were included from one academic center. The authors retrospectively studied the free-text operative notes of included patients for the primary outcome that was defined as intraoperative durotomy. Incidental durotomy occurred in 9% (93 of 1000), 8% (108 of 1279), and 6% (58 of 944) of the patients, respectively, in the Massachusetts, Maryland, and Australia cohorts. No missing reports were observed. Three datasets (Massachusetts, Australian, and combined Massachusetts and Australian) were divided into training and holdout test sets in an 80:20 ratio. An extreme gradient boosting (an efficient and flexible tree-based algorithm) NLP algorithm was individually trained on each training set, and the performance of the three NLP algorithms (respectively American, Australian, and combined) was assessed by discrimination via area under the receiver operating characteristic curves (AUC-ROC; this measures the model's ability to distinguish patients who obtained the outcomes from those who did not), calibration metrics (which plot the predicted and the observed probabilities) and Brier score (a composite of discrimination and calibration). In addition, the sensitivity (true positives, recall), specificity (true negatives), positive predictive value (also known as precision), negative predictive value, F1-score (composite of precision and recall), positive likelihood ratio, and negative likelihood ratio were calculated. The combined NLP algorithm (the combined Massachusetts and Australian data) achieved excellent performance on independent testing data from Australia (AUC-ROC 0.97 [95% confidence interval 0.87 to 0.99]), Massachusetts (AUC-ROC 0.99 [95% CI 0.80 to 0.99]) and Maryland (AUC-ROC 0.95 [95% CI 0.93 to 0.97]). The NLP developed based on the Massachusetts cohort had excellent performance in the Maryland cohort (AUC-ROC 0.97 [95% CI 0.95 to 0.99]) but worse performance in the Australian cohort (AUC-ROC 0.74 [95% CI 0.70 to 0.77]). We demonstrated the clinical utility and reproducibility of an NLP algorithm with combined datasets retaining excellent performance in individual countries relative to algorithms developed in the same country alone for detection of incidental durotomy. Further multi-institutional, international collaborations can facilitate the creation of universal NLP algorithms that improve the quality and safety of orthopaedic surgery globally. The combined NLP algorithm has been incorporated into a freely accessible web application that can be found at https://sorg-apps.shinyapps.io/nlp_incidental_durotomy/ . Clinicians and researchers can use the tool to help incorporate the model in evaluating spine registries or quality and safety departments to automate detection of incidental durotomy and optimize prevention efforts. Level III, diagnostic study.

Comparing Machine Learning to Regression Methods for Mortality Prediction Using Veterans Affairs Electronic Health Record Clinical Data.

It is unclear whether machine learning methods yield more accurate electronic health record (EHR) prediction models compared with traditional regression methods. The objective of this study was to compare machine learning and traditional regression models for 10-year mortality prediction using EHR data. This was a cohort study. Veterans Affairs (VA) EHR data. Veterans age above 50 with a primary care visit in 2005, divided into separate training and testing cohorts (n= 124,360 each). The primary outcome was 10-year all-cause mortality. We considered 924 potential predictors across a wide range of EHR data elements including demographics (3), vital signs (9), medication classes (399), disease diagnoses (293), laboratory results (71), and health care utilization (149). We compared discrimination (c-statistics), calibration metrics, and diagnostic test characteristics (sensitivity, specificity, and positive and negative predictive values) of machine learning and regression models. Our cohort mean age (SD) was 68.2 (10.5), 93.9% were male; 39.4% died within 10 years. Models yielded testing cohort c-statistics between 0.827 and 0.837. Utilizing all 924 predictors, the Gradient Boosting model yielded the highest c-statistic [0.837, 95% confidence interval (CI): 0.835-0.839]. The full (unselected) logistic regression model had the highest c-statistic of regression models (0.833, 95% CI: 0.830-0.835) but showed evidence of overfitting. The discrimination of the stepwise selection logistic model (101 predictors) was similar (0.832, 95% CI: 0.830-0.834) with minimal overfitting. All models were well-calibrated and had similar diagnostic test characteristics. Our results should be confirmed in non-VA EHRs. The differences in c-statistic between the best machine learning model (924-predictor Gradient Boosting) and 101-predictor stepwise logistic models for 10-year mortality prediction were modest, suggesting stepwise regression methods continue to be a reasonable method for VA EHR mortality prediction model development.

Abstract P021: Performance Of Pooled Cohort Equations And MESA Risk Score Across Race/Ethnicity And Socioeconomic Status To Estimate 10-year Cardiovascular Risk In Diverse New England Cohort

Introduction: The Pooled Cohort Equations (PCE) are the clinical standard for risk estimation of 10-year ASCVD risk, but they have not been extensively validated in minority populations or compared with newer risk estimation calculators such as the Multi-Ethnic Study of Atherosclerosis (MESA) 10-year CHD risk score. Methods: We collected 20-year electronic medical record data (1997-2017) from 34,138 patients in New England (age range in 2007: 55-75). These patients were not on statins and had no documented ASCVD event between 1997 and 2007. Cardiovascular risk factors were collected to calculate PCE and MESA (excluding coronary calcium) scores in 2007. 10-year incident ASCVD events were recorded using ICD codes (2007-17). Risk estimates were evaluated in the total population, race/ethnicity subgroups, and by socioeconomic status (SES) (quintiles of median community income). Results: Overall, PCE and MESA scores had similar discriminative performance (c-statistic: 0.69 vs. 0.70, p-value = 0.15). Both PCE and MESA scores had the best discriminative performance in whites and among individuals with top quintile SES relative to other racial and SES subgroups. In whites, MESA scores had better discriminative performance than PCE (c-statistic: 0.71 vs. 0.70, p-value = 0.005). PCE had higher discriminative performance for African Americans, Hispanics, and Asians but was not statistically significant. PCE and MESA scores had similar calibration metrics for whites but had differing degrees of calibration in African Americans (O to E: 0.79), Hispanics (O to E: 0.83), and Asians (O to E: 0.95) compared with MESA scores (O to E: African Americans: 1.17, Hispanics: 0.96, Asians: 1.51). Conclusion: While MESA and PCE have similar trends in discriminative performance across race/ethnicity and SES, they differed in calibration across minority groups. Clinicians should be cognizant of model performance differences across racial/ethnic and SES subgroups when using risk scores for determining preventive therapies such as statins.

Predicting the long-term cognitive trajectories using machine learning approaches: A Chinese nationwide longitudinal database

ObjectivesThis study aimed to explore the long-term cognitive trajectories and its’ determinants, and construct prediction models for identifying high-risk populations with unfavorable cognitive trajectories. MethodsThis study included 3502 older adults aged 65–105 years at their first observations in a 16-year longitudinal cohort study. Cognitive function was measured by the Chinese version Mini Mental State Examination. The heterogeneity of cognitive function was identified through mixed growth model. Machine learning algorithms, namely regularized logistic regression (r-LR), support vector machine (SVM), random forest (RF), and super learner (SL) were used to predict cognitive trajectories. Discrimination and calibration metrics were used for performance evaluation. ResultsTwo distinct trajectories were identified according to the changes of MMSE scores: intact cognitive functioning (93.6%), and dementia (6.4%). Older age, female gender, Han ethnicity, having no schooling, rural residents, low-frequency leisure activities, and low baseline BADL score were associated with a rapid decline in cognitive function. r-LR, SVM, and SL performed well in predicting cognitive trajectories (Sensitivity: 0.73, G-mean: 0.65). Age and psychological well-being were key predictors. ConclusionTwo cognitive trajectories were identified among older Chinese, and the identified key factors could be targeted for constructing early risk prediction models.

Graphical calibration curves and the integrated calibration index (ICI) for competing risk models

BackgroundAssessing calibration—the agreement between estimated risk and observed proportions—is an important component of deriving and validating clinical prediction models. Methods for assessing the calibration of prognostic models for use with competing risk data have received little attention.MethodsWe propose a method for graphically assessing the calibration of competing risk regression models. Our proposed method can be used to assess the calibration of any model for estimating incidence in the presence of competing risk (e.g., a Fine-Gray subdistribution hazard model; a combination of cause-specific hazard functions; or a random survival forest). Our method is based on using the Fine-Gray subdistribution hazard model to regress the cumulative incidence function of the cause-specific outcome of interest on the predicted outcome risk of the model whose calibration we want to assess. We provide modifications of the integrated calibration index (ICI), of E50 and of E90, which are numerical calibration metrics, for use with competing risk data. We conducted a series of Monte Carlo simulations to evaluate the performance of these calibration measures when the underlying model has been correctly specified and when the model was mis-specified and when the incidence of the cause-specific outcome differed between the derivation and validation samples. We illustrated the usefulness of calibration curves and the numerical calibration metrics by comparing the calibration of a Fine-Gray subdistribution hazards regression model with that of random survival forests for predicting cardiovascular mortality in patients hospitalized with heart failure.ResultsThe simulations indicated that the method for constructing graphical calibration curves and the associated calibration metrics performed as desired. We also demonstrated that the numerical calibration metrics can be used as optimization criteria when tuning machine learning methods for competing risk outcomes.ConclusionsThe calibration curves and numeric calibration metrics permit a comprehensive comparison of the calibration of different competing risk models.

Hydrological process knowledge in catchment modelling – Lessons and perspectives from 60 years development

AbstractHydrological process knowledge has advanced significantly during the past six decades. During the same period catchment models have undergone major developments including simple black box models, lumped conceptual models, hydrological response unit models, spatially distributed process‐based models and, recently, the emergence of machine learning hybrid models. This development has been enabled by improved understanding of hydrological processes together with ever increasing computer power and improved availability and accessibility of data. During the first couple of decades, a key assumption motivating the development towards increasing complexity of model codes was that more detailed process description would lead to more accurate model simulations and enable prediction of impacts from human activities that previous models were not able to provide. Subsequently, scientific tests showed that this is very often not the case, leading towards a recognition of the importance of careful model evaluation accounting for key uncertainties in data, model parameters and model conceptual understanding. We have reviewed 54 model studies from the past 60 years and characterized them with respect to model type, spatial discretization and model evaluation techniques. This showed clear development trends and different strategies for enhancing hydrological process knowledge in models. In addition, we present a case study, where we use two models for the same catchment. The models are identical except for the spatial discretization of 100 m and 500 m, respectively. The two models have an apparent equal performance measured against standard calibration metrics, but nevertheless show large differences when considering detailed process information such as partitioning of streamflow components and water table depth patterns, that was not considered during the model calibration process. The paper discusses perspectives for enhancing hydrological process knowledge in future catchment modelling concluding that the emergence of big data is likely to become a major game changer.

Impacts of spatiotemporal resolution and tiling on SLEUTH model calibration and forecasting for urban areas with unregulated growth patterns

ABSTRACT The SLEUTH model provides a framework for understanding land use evolution around urban areas. Calibration of SLEUTH’s behavioral coefficients can be impacted by scale and nonlinear transitions due to the SLEUTH land use deltatron module’s assumption of linear Markov change probabilities. This study attempted to establish what spatial resolution and temporal scale produces the most accurate forecasts given the linear change assumption. The impact of tiling the input data was also examined. To determine these, SLEUTH was calibrated at four spatial and three temporal scales for Ibadan, Nigeria using both untiled and tiled data. Calibration results were evaluated using accuracy metrics including Figure of Merit (FOM) and mean uncertainty. The best mix of calibration metrics (FOM 0.26) and mean uncertainty (11.64) was achieved at 30 m resolution and an intermediate temporal interval. Tiling input data led to overfitting, allowing good model fit within individual tiles but a reduction in trend recognition across land use types. Subsequently, a 2040 projection that is as accurate as possible, and scientifically justifiable given the available data, was produced. The findings provide a framework for understanding the effect of spatiotemporal scale on SLEUTH inputs that require tiling particularly for urban areas in the global south.

Development and Validation of a Predictive Model to Identify Patients With an Ascending Thoracic Aortic Aneurysm.

BackgroundScreening protocols do not exist for ascending thoracic aortic aneurysms (ATAAs). A risk prediction algorithm may aid targeted screening of patients with an undiagnosed ATAA to prevent aortic dissection. We aimed to develop and validate a risk model to identify those at increased risk of having an ATAA, based on readily available clinical information.Methods and ResultsThis is a cross‐sectional study of computed tomography scans involving the chest at a tertiary care center on unique patients aged 50 to 85 years between 2013 and 2016. These criteria yielded 21 325 computed tomography scans. The double‐oblique technique was used to measure the ascending thoracic aorta, and an ATAA was defined as >40 mm in diameter. A logistic regression model was fitted for the risk of ATAA, with readily available demographics and comorbidity variables. Model performance was characterized by discrimination and calibration metrics via split‐sample testing. Among the 21 325 patients, there were 560 (2.6%) patients with an ATAA. The multivariable model demonstrated that older age, higher body surface area, history of arrhythmia, aortic valve disease, hypertension, and family history of aortic aneurysm were associated with increased risk of an ATAA, whereas female sex and diabetes were associated with a lower risk of an ATAA. The C statistic of the model was 0.723±0.016. The regression coefficients were transformed to scores that allow for point‐of‐care calculation of patients' risk.ConclusionsWe developed and internally validated a model to predict patients' risk of having an ATAA based on demographic and clinical characteristics. This algorithm may guide the targeted screening of an undiagnosed ATAA.

Supplementing Existing Societal Risk Models for Surgical Aortic Valve Replacement With Machine Learning for Improved Prediction.

BackgroundThis study evaluated the role of supplementing Society of Thoracic Surgeons (STS) risk models for surgical aortic valve replacement with machine learning (ML).Methods and ResultsAdults undergoing isolated surgical aortic valve replacement in the STS National Database between 2007 and 2017 were included. ML models for operative mortality and major morbidity were previously developed using extreme gradient boosting. Concordance and discordance in predicted risk between ML and STS models were defined using equal‐size tertile‐based thresholds of risk. Calibration metrics and discriminatory capability were compared between concordant and discordant patients. A total of 243 142 patients were included. Nearly all calibration metrics were improved in cases of concordance. Similarly, concordance indices improved substantially in cases of concordance for all models with the exception of deep sternal wound infection. The greatest improvements in concordant versus discordant cases were in renal failure: ML model (concordance index, 0.660 [95% CI, 0.632–0.687] discordant versus 0.808 [95% CI, 0.794–0.822] concordant) and STS model (concordance index, 0.573 [95% CI, 0.549–0.576] discordant versus 0.797 [95% CI, 0.782–0.811] concordant) (each P<0.001). Excluding deep sternal wound infection, the concordance indices ranged from 0.549 to 0.660 for discordant cases and 0.674 to 0.808 for concordant cases.ConclusionsSupplementing ML models with existing STS models for surgical aortic valve replacement may have an important role in risk prediction and should be explored further. In particular, for the roughly 25% to 50% of patients demonstrating discordance in estimated risk between ML and STS, there appears to be a substantial decline in predictive performance suggesting vulnerability of the existing models in these patient subsets.

Machine learning-based clinical outcome prediction in surgery for acromegaly

PurposeBiochemical remission (BR), gross total resection (GTR), and intraoperative cerebrospinal fluid (CSF) leaks are important metrics in transsphenoidal surgery for acromegaly, and prediction of their likelihood using machine learning would be clinically advantageous. We aim to develop and externally validate clinical prediction models for outcomes after transsphenoidal surgery for acromegaly.MethodsUsing data from two registries, we develop and externally validate machine learning models for GTR, BR, and CSF leaks after endoscopic transsphenoidal surgery in acromegalic patients. For the model development a registry from Bologna, Italy was used. External validation was then performed using data from Zurich, Switzerland. Gender, age, prior surgery, as well as Hardy and Knosp classification were used as input features. Discrimination and calibration metrics were assessed.ResultsThe derivation cohort consisted of 307 patients (43.3% male; mean [SD] age, 47.2 [12.7] years). GTR was achieved in 226 (73.6%) and BR in 245 (79.8%) patients. In the external validation cohort with 46 patients, 31 (75.6%) achieved GTR and 31 (77.5%) achieved BR. Area under the curve (AUC) at external validation was 0.75 (95% confidence interval: 0.59–0.88) for GTR, 0.63 (0.40–0.82) for BR, as well as 0.77 (0.62–0.91) for intraoperative CSF leaks. While prior surgery was the most important variable for prediction of GTR, age, and Hardy grading contributed most to the predictions of BR and CSF leaks, respectively.ConclusionsGross total resection, biochemical remission, and CSF leaks remain hard to predict, but machine learning offers potential in helping to tailor surgical therapy. We demonstrate the feasibility of developing and externally validating clinical prediction models for these outcomes after surgery for acromegaly and lay the groundwork for development of a multicenter model with more robust generalization.

Cardiovascular risk assessment in people living with HIV compared to the general population

Abstract Background Effective cardiovascular preventive strategies are crucial among people living with HIV (PLWH), who are facing a high burden of atherosclerotic cardiovascular disease (ASCVD). However, it remains unclear which cardiovascular risk score is the most appropriate in clinical practice. Purpose We aimed to prospectively assess and compare the accuracy of widely used cardiovascular risk scores in PLWH and individuals from the general population. Methods We used data from the Swiss HIV Cohort Study (SHCS), a longitudinal study involving 20,802 HIV-infected adults aged over 18 years, and from the CoLaus|PsyCoLaus study, a Swiss population-based cohort including 6,733 individuals aged 35–75 years. The European Systematic Coronary Risk Evaluation Score (SCORE), the North American Pooled Cohort Equation (PCE) and the HIV-specific Data Collection o-n Adverse events of Anti-HIV Drugs (D:A:D) score were calculated for all participants free from ASCVD between January 1, 2003 and December 31, 2009. Accuracy of the scores was assessed based on discrimination and calibration metrics for each cohort separately using incident ASCVD as outcome. The value of adding HIV-specific factors to the model presenting the best predictive capacities between SCORE and PCE was evaluated using the net reclassification index (NRI). Results 6,373 PLWH (28.4% women; aged 40.6 [SD, 9.9]; 57.2% on antiretroviral therapy) and 5,403 individuals from the general population (53.5% women, aged 52.8 [SD, 10.7]) were included in the analysis with a mean follow-up time of 13.5 (SD, 4.1) and 9.9 (SD, 2.3) years, respectively. 533 (8.4%) participants in the SHCS and 374 (6.9%) in the CoLaus|PsyCoLaus study experienced an incident ASCVD translating into age-adjusted incidence rates of 12.9 vs. 7.5 per 1,000 person-year, respectively. In SHCS, PCE and D:A:D presented discriminative capacities with AUROC of 0.757 (95% CI, 0.736–0.777) and 0.763 (95% CI, 0.743–0.783), respectively, compared to SCORE (0.704 [95% CI, 0.681–0.728]). Calibration of all scores was suboptimal in SHCS, with under-prediction of ASCVD in the higher deciles of risk compared to the CoLaus|PsyCoLaus study. Adding CD4 nadir (&amp;lt;200 cells/mm3) and abacavir exposure as categorical variables to PCE resulted in a marginal improvement in discrimination and in a global NRI of 2.7% (95% CI, 0.3–5.1, p-value = 0.03). Conclusions PLWH presented a two-fold higher rate of incident ASCVD compared to individuals of the same age from the general population. The accuracy of PCE score to predict ASCVD in PLWH is equivalent to the D:A:D score and may represent a better alternative due to its reduced set of variables and its widespread use. Adding HIV-specific factors to PCE did not improve its predictive performance. Funding Acknowledgement Type of funding sources: Public grant(s) – National budget only. Main funding source(s): Swiss National Science Foundation

Prediction of Major Adverse Events After Endovascular Aneurysm Repair Using a Machine Learning Model

Predicting major adverse events (MAE), including type I and III endoleaks (ELs), after endovascular aneurysm repair (EVAR) is essential for clinical decision-making. No clinical decision instruments can reliably predict for MAE after EVAR. We developed a machine learning model (MLM) for predicting MAE within 3 years after elective EVAR using preoperative demographic information and aneurysm geometry. MLMs infer complex interaction without prespecification, possibly increasing the prediction accuracy compared with logistic regression models (LRMs). Patients with intact infrarenal abdominal aortic aneurysms who had undergone EVAR with preoperative M2S reconstruction from 2010 to 2017 at a single institution were analyzed. The MAEs included aneurysm- or intervention-related death within 30 days, aneurysm rupture, the development of type I or III ELs, greater than three reinterventions, or EVAR explantation. Patients with <3 years of follow-up were excluded. Seventy-four preoperative predictors were evaluated. Of these, 30 were geometric factors derived from M2S models. The remaining factors were demographic. The performance of LRMs, random forest models (RFMs), and gradient-boosted models (GBMs) were evaluated using a 75% and 25% training and testing split, fivefold cross-validation, and receiver operating characteristic performance metric. A total of 250 patients were included (mean age, 71.9 ± 8.6 years; 86% male). The abdominal aortic aneurysm diameter was 56.0 ± 10.6 mm, and the infrarenal neck length was 24.6 ± 4.5 mm. Of the 250 patients, 40 (16%) had experienced MAE: 11 had died within 30 days, 3 had experienced aneurysm rupture, 10 had developed a type I EL, 10 had developed a type III EL, 10 had required graft explantation, and 4 had undergone four or more reinterventions. The most significant 45 predictive variables are shown in the Fig. All discrimination and calibration metrics favored the MLM over the LRM. The area under the curve for the LRM was 0.70 compared with 0.83 for the RFM and 0.77 for the GBM. The LRM classification accuracy was 68% compared with 84% for the RFM and GBM. The GBM showed superior calibration performance (Hosmer-Lemeshow P = .201). The removal of the geometric parameters significantly degraded the performance for all the models (likelihood ratio P < .001). The MLM displayed superior discrimination and calibration compared with the LRM in predicting MAE for the first 3 years after elective EVAR, possibly owing to the complex interaction terms in the predictive factors. The anatomic characteristics were more influential than the demographic data in predicting for MAE. Our MLM might predict for MAE after EVAR and thereby aid in selecting patients for EVAR, nonoperative observation, or open repair.

Jointly Calibrating Hydrologic Model Parameters and State Adjustments

AbstractA method is presented to address model state uncertainty in hydrologic model simulation. This is achieved by introducing tuneable parameters that allow adjustments to the model states. Excessive dimensionality is avoided by introducing only a limited number of parameters that control the index (timing) and size of the state adjustments. The method is designed to compensate for issues with hydrologic model structures, particularly those relevant to the soil moisture state in a rainfall‐runoff model. In the context of water resource planning and management, errors in the model states have often been overlooked as an important source of uncertainty and have the potential to significantly degrade model simulations. A synthetic study shows that a classical parameter estimation approach will produce biased distributions when state errors exist, and that the proposed state and parameter uncertainty estimation (SPUE) can remove the bias in parameter estimates for improved model simulations. In a real case study, SPUE and the classical approach are implemented in 46 sites around Australia. The results show that hydrologic parameter distributions for a selected conceptual model can be significantly different when accounting for state uncertainty. This has large implications for scenario modeling since it puts into dispute how to determine appropriate parameters for such studies. SPUE outperforms the classical approach in a range of calibration and validation metrics, particularly for sites that contain zero flows. Future work involves testing SPUE with different hydrologic models and likelihood formulations, and enhancing rigor by explicitly accounting for observational data uncertainty.

Want to model a species niche? A step-by-step guideline on correlative ecological niche modelling

The use of correlative ecological niche models has highly increased in the last decade. Despite all literature and textbooks in this field, few practical guidelines exist on the correct application of these techniques. We present here a step-by-step guideline explaining best practices for calculating correlative ecological niche models considering their conceptual and statistical assumptions and limitations. We divided the modelling process into four stages: 1) data collection and preparation; 2) model calculation; 3) model evaluation and validation; 4) and model application. Based on ecological niche theory, we review the concepts of ecological niche and how they can be modelled; classes of correlative models; modelling software; selection of study area; data sources for species records and environmental variables; types of species records and their influence on correlative models; errors in species records; minimum number of species records and environmental variables; effects of prevalence, sampling design, biases, and collinearity between variables; model calculation; model projection to different scenarios in time and space; ensemble modelling; model validation; classification, discrimination and calibration metrics; calculation of null models; analysis of model results; and model thresholding. This guideline is expected to help potential users to obtain better results when using correlative ecological niche models.

Numerical Modeling of the Effects of Toe Configuration on Throughflow in Rockfill Dams

The rockfill toe structure situated within the downstream slope of rockfill dams is an integral part of a defense mechanism safeguarding the dam structure in throughflow situations. Recent studies have concluded that the rockfill toe structure can have significant impacts on throughflow development and stability of rockfill dams under scenarios of accidental throughflow caused by overtopping of the dam core. The ability to numerically model the effect of various toe configurations on flow through rockfill dams can support the design of effective toe drainage structures for rockfill dams. Development and calibration of a reliable numerical modeling tool in this regard has been challenging owing to lack of availability of extensive datasets from physical modeling investigations. This study further employs datasets gathered by a recent physical modeling study investigating the effects of various toe configurations on throughflow development in rockfill dam models. A commercial numerical seepage modeling tool with an option for non-Darcy flow was calibrated against the datasets with good calibration metrics. The study is novel in providing a rare report on the usage of this option. The calibrated tool can further be employed to carry out a wide array of simulations to arrive at an ideal design for a toe structure for rockfill dams and for assessment of hydraulic performance of toe structures.

Class-wise Calibration: A Case Study on COVID-19 Hate Speech

Proper calibration of deep-learning models is critical for many high-stakes problems. In this paper, we show that existing calibration metrics fail to pay attention to miscalibration on individual classes, hence overlooking minority classes and causing significant issues on imbalanced classification problems. Using a COVID-19 hate-speech dataset, we first discover that in imbalanced datasets, miscalibration error on an individual class varies greatly, and error on minority classes can be magnitude times worse than what is suggested by the overall calibration performance. To address this issue, we propose a new metric based on expected miscalibration error, named as Contraharmonic Expected Calibration Error (CECE), which punishes severe miscalibration on individual classes. We further devise a novel variant of temperature scaling for imbalanced data to improve class-wise miscalibration, which re-weights the loss function by the inverse class count to tune the scaling parameter to reduce worst-case minority calibration error. Our case study on a benchmarking COVID-19 hate speech task shows the effectiveness of our calibration metric and our temperature scaling strategy.

Toward Dynamic Risk Prediction of Outcomes After Coronary Artery Bypass Graft: Improving Risk Prediction With Intraoperative Events Using Gradient Boosting.

Intraoperative data may improve models predicting postoperative events. We evaluated the effect of incorporating intraoperative variables to the existing preoperative model on the predictive performance of the model for coronary artery bypass graft. We analyzed 378 572 isolated coronary artery bypass graft cases performed across 1083 centers, using the national Society of Thoracic Surgeons Adult Cardiac Surgery Database between 2014 and 2016. Outcomes were operative mortality, 5 postoperative complications, and composite representation of all events. We fitted models by logistic regression or extreme gradient boosting (XGBoost). For each modeling approach, we used preoperative only, intraoperative only, or pre+intraoperative variables. We developed 84 models with unique combinations of the 3 variable sets, 2 variable selection methods, 2 modeling approaches, and 7 outcomes. Each model was tested in 20 iterations of 70:30 stratified random splitting into development/testing samples. Model performances were evaluated on the testing dataset using the C statistic, area under the precision-recall curve, and calibration metrics, including the Brier score. The mean patient age was 65.3 years, and 24.7% were women. Operative mortality, excluding intraoperative death, occurred in 1.9%. In all outcomes, models that considered pre+intraoperative variables demonstrated significantly improved Brier score and area under the precision-recall curve compared with models considering pre or intraoperative variables alone. XGBoost without external variable selection had the best C statistics, Brier score, and area under the precision-recall curve values in 4 of the 7 outcomes (mortality, renal failure, prolonged ventilation, and composite) compared with logistic regression models with or without variable selection. Based on the calibration plots, risk restratification for mortality showed that the logistic regression model underestimated the risk in 11 114 patients (9.8%) and overestimated in 12 005 patients (10.6%). In contrast, the XGBoost model underestimated the risk in 7218 patients (6.4%) and overestimated in 0 patients (0%). In isolated coronary artery bypass graft, adding intraoperative variables to preoperative variables resulted in improved predictions of all 7 outcomes. Risk models based on XGBoost may provide a better prediction of adverse events to guide clinical care.

An Improvement of Survival Stratification in Glioblastoma Patients via Combining Subregional Radiomics Signatures.

PurposeTo investigate whether combining multiple radiomics signatures derived from the subregions of glioblastoma (GBM) can improve survival prediction of patients with GBM.MethodsIn total, 129 patients were included in this study and split into training (n = 99) and test (n = 30) cohorts. Radiomics features were extracted from each tumor region then radiomics scores were obtained separately using least absolute shrinkage and selection operator (LASSO) COX regression. A clinical nomogram was also constructed using various clinical risk factors. Radiomics nomograms were constructed by combing a single radiomics signature from the whole tumor region with clinical risk factors or combining three radiomics signatures from three tumor subregions with clinical risk factors. The performance of these models was assessed by the discrimination, calibration and clinical usefulness metrics, and was compared with that of the clinical nomogram.ResultsIncorporating the three radiomics signatures, i.e., Radscores for ET, NET, and ED, into the radiomics-based nomogram improved the performance in estimating survival (C-index: training/test cohort: 0.717/0.655) compared with that of the clinical nomogram (C-index: training/test cohort: 0.633/0.560) and that of the radiomics nomogram based on single region radiomics signatures (C-index: training/test cohort: 0.656/0.535).ConclusionThe multiregional radiomics nomogram exhibited a favorable survival stratification accuracy.

Performance assessment of the metastatic spinal tumor frailty index using machine learning algorithms: limitations and future directions.

Frailty is recognized as an important consideration in patients with cancer who are undergoing therapies, including spine surgery. The definition of frailty in the context of spinal metastases is unclear, and few have studied such markers and their association with postoperative outcomes and survival. Using national databases, the metastatic spinal tumor frailty index (MSTFI) was developed as a tool to predict outcomes in this specific patient population and has not been tested with external data. The purpose of this study was to test the performance of the MSTFI with institutional data and determine whether machine learning methods could better identify measures of frailty as predictors of outcomes. Electronic health record data from 479 adult patients admitted to the Massachusetts General Hospital for metastatic spinal tumor surgery from 2010 to 2019 formed a validation cohort for the MSTFI to predict major complications, in-hospital mortality, and length of stay (LOS). The 9 parameters of the MSTFI were modeled in 3 machine learning algorithms (lasso regularization logistic regression, random forest, and gradient-boosted decision tree) to assess clinical outcome prediction and determine variable importance. Prediction performance of the models was measured by computing areas under the receiver operating characteristic curve (AUROCs), calibration, and confusion matrix metrics (positive predictive value, sensitivity, and specificity) and was subjected to internal bootstrap validation. Of 479 patients (median age 64 years [IQR 55-71 years]; 58.7% male), 28.4% had complications after spine surgery. The in-hospital mortality rate was 1.9%, and the mean LOS was 7.8 days. The MSTFI demonstrated poor discrimination for predicting complications (AUROC 0.56, 95% CI 0.50-0.62) and in-hospital mortality (AUROC 0.69, 95% CI 0.54-0.85) in the validation cohort. For postoperative complications, machine learning approaches showed a greater advantage over the logistic regression model used to develop the MSTFI (AUROC 0.62, 95% CI 0.56-0.68 for random forest vs AUROC 0.56, 95% CI 0.50-0.62 for logistic regression). The random forest model had the highest positive predictive value (0.53, 95% CI 0.43-0.64) and the highest negative predictive value (0.77, 95% CI 0.72-0.81), with chronic lung disease, coagulopathy, anemia, and malnutrition identified as the most important predictors of postoperative complications. This study highlights the challenges of defining and quantifying frailty in the metastatic spine tumor population. Further study is required to improve the determination of surgical frailty in this specific cohort.

Using machine learning to improve risk prediction in durable left ventricular assist devices.

Risk models have historically displayed only moderate predictive performance in estimating mortality risk in left ventricular assist device therapy. This study evaluated whether machine learning can improve risk prediction for left ventricular assist devices. Primary durable left ventricular assist devices reported in the Interagency Registry for Mechanically Assisted Circulatory Support between March 1, 2006 and December 31, 2016 were included. The study cohort was randomly divided 3:1 into training and testing sets. Logistic regression and machine learning models (extreme gradient boosting) were created in the training set for 90-day and 1-year mortality and their performance was evaluated after bootstrapping with 1000 replications in the testing set. Differences in model performance were also evaluated in cases of concordance versus discordance in predicted risk between logistic regression and extreme gradient boosting as defined by equal size patient tertiles. A total of 16,120 patients were included. Calibration metrics were comparable between logistic regression and extreme gradient boosting. C-index was improved with extreme gradient boosting (90-day: 0.707 [0.683–0.730] versus 0.740 [0.717–0.762] and 1-year: 0.691 [0.673–0.710] versus 0.714 [0.695–0.734]; each p<0.001). Net reclassification index analysis similarly demonstrated an improvement of 48.8% and 36.9% for 90-day and 1-year mortality, respectively, with extreme gradient boosting (each p<0.001). Concordance in predicted risk between logistic regression and extreme gradient boosting resulted in substantially improved c-index for both logistic regression and extreme gradient boosting (90-day logistic regression 0.536 versus 0.752, 1-year logistic regression 0.555 versus 0.726, 90-day extreme gradient boosting 0.623 versus 0.772, 1-year extreme gradient boosting 0.613 versus 0.742, each p<0.001). These results demonstrate that machine learning can improve risk model performance for durable left ventricular assist devices, both independently and as an adjunct to logistic regression.