Performance Of Clinical Prediction Models Research Articles

Impact of random oversampling and random undersampling on the performance of prediction models developed using observational health data

BackgroundThere is currently no consensus on the impact of class imbalance methods on the performance of clinical prediction models. We aimed to empirically investigate the impact of random oversampling and random undersampling, two commonly used class imbalance methods, on the internal and external validation performance of prediction models developed using observational health data.MethodsWe developed and externally validated prediction models for various outcomes of interest within a target population of people with pharmaceutically treated depression across four large observational health databases. We used three different classifiers (lasso logistic regression, random forest, XGBoost) and varied the target imbalance ratio. We evaluated the impact on model performance in terms of discrimination and calibration. Discrimination was assessed using the area under the receiver operating characteristic curve (AUROC) and calibration was assessed using calibration plots.ResultsWe developed and externally validated a total of 1,566 prediction models. On internal and external validation, random oversampling and random undersampling generally did not result in higher AUROCs. Moreover, we found overestimated risks, although this miscalibration could largely be corrected by recalibrating the models towards the imbalance ratios in the original dataset.ConclusionsOverall, we found that random oversampling or random undersampling generally does not improve the internal and external validation performance of prediction models developed in large observational health databases. Based on our findings, we do not recommend applying random oversampling or random undersampling when developing prediction models in large observational health databases.

Follow-up ASPECTS improves prediction of potentially lethal malignant edema in patients with large middle cerebral artery stroke

BackgroundRecent studies have shown that follow-up head CT is a strong predictor of functional outcomes in patients with middle cerebral artery stroke and mechanical thrombectomy. We sought to determine whether...

Dynamic updating of clinical survival prediction models in a changing environment

BackgroundOver time, the performance of clinical prediction models may deteriorate due to changes in clinical management, data quality, disease risk and/or patient mix. Such prediction models must be updated in order to remain useful. In this study, we investigate dynamic model updating of clinical survival prediction models. In contrast to discrete or one-time updating, dynamic updating refers to a repeated process for updating a prediction model with new data. We aim to extend previous research which focused largely on binary outcome prediction models by concentrating on time-to-event outcomes. We were motivated by the rapidly changing environment seen during the COVID-19 pandemic where mortality rates changed over time and new treatments and vaccines were introduced.MethodsWe illustrate three methods for dynamic model updating: Bayesian dynamic updating, recalibration, and full refitting. We use a simulation study to compare performance in a range of scenarios including changing mortality rates, predictors with low prevalence and the introduction of a new treatment. Next, the updating strategies were applied to a model for predicting 70-day COVID-19-related mortality using patient data from QResearch, an electronic health records database from general practices in the UK.ResultsIn simulated scenarios with mortality rates changing over time, all updating methods resulted in better calibration than not updating. Moreover, dynamic updating outperformed ad hoc updating. In the simulation scenario with a new predictor and a small updating dataset, Bayesian updating improved the C-index over not updating and refitting. In the motivating example with a rare outcome, no single updating method offered the best performance.ConclusionsWe found that a dynamic updating process outperformed one-time discrete updating in the simulations. Bayesian updating offered good performance overall, even in scenarios with new predictors and few events. Intercept recalibration was effective in scenarios with smaller sample size and changing baseline hazard. Refitting performance depended on sample size and produced abrupt changes in hazard ratio estimates between periods.

Imputation and missing indicators for handling missing data in the development and deployment of clinical prediction models: A simulation study.

Background: In clinical prediction modelling, missing data can occur at any stage of the model pipeline; development, validation or deployment. Multiple imputation is often recommended yet challenging to apply at deployment; for example, the outcome cannot be in the imputation model, as recommended under multiple imputation. Regression imputation uses a fitted model to impute the predicted value of missing predictors from observed data, and could offer a pragmatic alternative at deployment. Moreover, the use of missing indicators has been proposed to handle informative missingness, but it is currently unknown how well this method performs in the context of clinical prediction models. Methods: We simulated data under various missing data mechanisms to compare the predictive performance of clinical prediction models developed using both imputation methods. We consider deployment scenarios where missing data is permitted or prohibited, imputation models that use or omit the outcome, and clinical prediction models that include or omit missing indicators. We assume that the missingness mechanism remains constant across the model pipeline. We also apply the proposed strategies to critical care data. Results: With complete data available at deployment, our findings were in line with existing recommendations; that the outcome should be used to impute development data when using multiple imputation and omitted under regression imputation. When missingness is allowed at deployment, omitting the outcome from the imputation model at the development was preferred. Missing indicators improved model performance in many cases but can be harmful under outcome-dependent missingness. Conclusion: We provide evidence that commonly taught principles of handling missing data via multiple imputation may not apply to clinical prediction models, particularly when data can be missing at deployment. We observed comparable predictive performance under multiple imputation and regression imputation. The performance of the missing data handling method must be evaluated on a study-by-study basis, and the most appropriate strategy for handling missing data at development should consider whether missing data are allowed at deployment. Some guidance is provided.

EHR foundation models improve robustness in the presence of temporal distribution shift

Temporal distribution shift negatively impacts the performance of clinical prediction models over time. Pretraining foundation models using self-supervised learning on electronic health records (EHR) may be effective in acquiring informative global patterns that can improve the robustness of task-specific models. The objective was to evaluate the utility of EHR foundation models in improving the in-distribution (ID) and out-of-distribution (OOD) performance of clinical prediction models. Transformer- and gated recurrent unit-based foundation models were pretrained on EHR of up to 1.8 M patients (382 M coded events) collected within pre-determined year groups (e.g., 2009–2012) and were subsequently used to construct patient representations for patients admitted to inpatient units. These representations were used to train logistic regression models to predict hospital mortality, long length of stay, 30-day readmission, and ICU admission. We compared our EHR foundation models with baseline logistic regression models learned on count-based representations (count-LR) in ID and OOD year groups. Performance was measured using area-under-the-receiver-operating-characteristic curve (AUROC), area-under-the-precision-recall curve, and absolute calibration error. Both transformer and recurrent-based foundation models generally showed better ID and OOD discrimination relative to count-LR and often exhibited less decay in tasks where there is observable degradation of discrimination performance (average AUROC decay of 3% for transformer-based foundation model vs. 7% for count-LR after 5–9 years). In addition, the performance and robustness of transformer-based foundation models continued to improve as pretraining set size increased. These results suggest that pretraining EHR foundation models at scale is a useful approach for developing clinical prediction models that perform well in the presence of temporal distribution shift.

Prognosis prediction performs better in patients with non-cirrhosis hepatitis B virus-related acute-on-chronic liver failure than those with cirrhosis.

The accurate prediction of the outcome of hepatitis B virus-related acute-on-chronic liver failure (HBV-ACLF) is impeded by population heterogeneity. The study aimed to assess the impact of underlying cirrhosis on the performance of clinical prediction models (CPMs). Using data from two multicenter, prospective cohorts of patients with HBV-ACLF, the discrimination, calibration, and clinical benefit were assessed for CPMs predicting 28-day and 90-day outcomes in patients with cirrhosis and those without, respectively. A total of 919 patients with HBV-ACLF were identified by Chinese Group on the Study of Severe Hepatitis B (COSSH) criteria, including 675 with cirrhosis and 244 without. COSSH-ACLF IIs, COSSH-ACLFs, Chronic Liver Failure-Consortium Acute-on-Chronic Liver Failure score (CLIF-C ACLFs), Tongji Prognostic Predictor Model score (TPPMs), Model for End-Stage Liver Disease score (MELDs), and MELD-Sodium score (MELD-Nas) were all strong predictors of short-term mortality in patients with HBV-ACLF. In contrast to a high model discriminative capacity in ACLF without cirrhosis, each prognostic model represents a marked decline of C-index, net reclassification index (NRI), and integrated discrimination improvement (IDI) in predicting either 28-day or 90-day prognosis of patients with cirrhosis. The hazard analysis identified largely overlapping risk factors of poor outcomes in both subgroups, while serum bilirubin was specifically associated with short-term mortality in patients with cirrhosis and blood urea nitrogen in patients without cirrhosis. A subgroup analysis in patients with cirrhosis showed a decline of discrimination of CPMS in those with ascites or infections compared to that in those without. Predicting the short-term outcome of HBV-ACLF by CPMs is optimal in patients without cirrhosis but limited in those with cirrhosis, at least partially due to the complicated ascites or infections.

External validation: a simulation study to compare cross-validation versus holdout or external testing to assess the performance of clinical prediction models using PET data from DLBCL patients

AimClinical prediction models need to be validated. In this study, we used simulation data to compare various internal and external validation approaches to validate models.MethodsData of 500 patients were simulated using distributions of metabolic tumor volume, standardized uptake value, the maximal distance between the largest lesion and another lesion, WHO performance status and age of 296 diffuse large B cell lymphoma patients. These data were used to predict progression after 2 years based on an existing logistic regression model. Using the simulated data, we applied cross-validation, bootstrapping and holdout (n = 100). We simulated new external datasets (n = 100, n = 200, n = 500) and simulated stage-specific external datasets (1), varied the cut-off for high-risk patients (2) and the false positive and false negative rates (3) and simulated a dataset with EARL2 characteristics (4). All internal and external simulations were repeated 100 times. Model performance was expressed as the cross-validated area under the curve (CV-AUC ± SD) and calibration slope.ResultsThe cross-validation (0.71 ± 0.06) and holdout (0.70 ± 0.07) resulted in comparable model performances, but the model had a higher uncertainty using a holdout set. Bootstrapping resulted in a CV-AUC of 0.67 ± 0.02. The calibration slope was comparable for these internal validation approaches. Increasing the size of the test set resulted in more precise CV-AUC estimates and smaller SD for the calibration slope. For test datasets with different stages, the CV-AUC increased as Ann Arbor stages increased. As expected, changing the cut-off for high risk and false positive- and negative rates influenced the model performance, which is clearly shown by the low calibration slope. The EARL2 dataset resulted in similar model performance and precision, but calibration slope indicated overfitting.ConclusionIn case of small datasets, it is not advisable to use a holdout or a very small external dataset with similar characteristics. A single small testing dataset suffers from a large uncertainty. Therefore, repeated CV using the full training dataset is preferred instead. Our simulations also demonstrated that it is important to consider the impact of differences in patient population between training and test data, which may ask for adjustment or stratification of relevant variables.

The thrombodynamic ratio as a predictor of 28-day mortality in sepsis patients

BackgroundThe thrombodynamic ratio (TDR) as a composite thromboelastography (TEG) parameter, has been proven to be valuable in multiple diseases. However, the association between TDR and mortality in sepsis has not been studied. MethodsOne hundred forty-one patients were enrolled in this retrospectively study. TEG was performed immediately at admission. Two cox proportional hazards models were developed for the prediction of 28-day mortality. The C statistic, continuous net reclassification index (cNRI) and integrated discriminatory index (IDI) were calculated to compare the discrimination performance of clinical models with and without the TDR value. The integrated calibration index (ICI) and E50 were calculated to compare the calibration. ResultsPatients with lower TDR were more likely to have organ impairments and increased 28-day mortality. The TDR value improved discrimination performance in both Model 1 (C statistic, 0.745 vs 0.735; cNRI 19.4%, p = 0.044; IDI 5.6%, p = 0.012) and Model 2 (C statistic, 0.761 vs 0.751; IDI, 5.1%, p = 0.012). Compared to the calibration curve of Model 1 without TDR, addition of TDR displayed better calibration (ICI, 0.023; E50, 0.021). ConclusionTDR value significantly predicts 28-day mortality in patients with sepsis and could improve the discrimination and calibration performance of clinical prediction models.

Clinical predictors of pulmonary tuberculosis among South African adults with HIV.

SummaryBackgroundTuberculosis (TB) clinical prediction rules rely on presence of symptoms, however many undiagnosed cases in the community are asymptomatic. This study aimed to explore the utility of clinical factors in predicting TB among people with HIV not seeking care.MethodsBaseline data were analysed from an observational cohort of ambulant adults with HIV in South Africa. Participants were tested for Mycobacterium tuberculosis (Mtb) sensitisation (interferon-γ release assay, IGRA) and microbiologically-confirmed prevalent pulmonary TB disease at baseline, and actively surveilled for incident TB through 15 months. Multivariable LASSO regression with post-selection inference was used to test associations with Mtb sensitisation and TB disease.FindingsBetween March 22, 2017, and May 15, 2018, 861 participants were enrolled; Among 851 participants included in the analysis, 94·5% were asymptomatic and 45·9% sensitised to Mtb. TB prevalence was 2·0% at baseline and incidence 2·3/100 person-years through 15 months follow-up. Study site was associated with baseline Mtb sensitisation (p < 0·001), prevalent (p < 0·001), and incident TB disease (p = 0·037). Independent of site, higher CD4 counts (per 50 cells/mm3, aOR 1·48, 95%CI 1·12–1·77, p = 0·006) were associated with increased IGRA positivity, and participants without TB disease (aOR 0·80, 95%CI 0·69–0·94, p = 0·006) had reduced IGRA positivity; no variables were independently associated with prevalent TB. Mixed ancestry (aHR 1·49, 95%CI 1·30–>1000, p = 0·005) and antiretroviral initiation (aHR 1·48, 95%CI 1·01–929·93, p = 0·023) were independently associated with incident TB. Models incorporating clinical features alone performed poorly in diagnosing prevalent (AUC 0·65, 95%CI 0·44–0·85) or predicting progression to incident (0·67, 0·46–0·88) TB.InterpretationCD4 count and antiretroviral initiation, proxies for immune status and HIV stage, were associated with Mtb sensitisation and TB disease. Inadequate performance of clinical prediction models may reflect predominantly subclinical disease diagnosed in this setting and unmeasured local site factors affecting transmission and progression.FundingThe CORTIS-HR study was funded by the Bill & Melinda Gates Foundation (OPP1151915) and by the Strategic Health Innovation Partnerships Unit of the South African Medical Research Council with funds received from the South African Department of Science and Technology. The regulatory sponsor was the University of Cape Town.

Updating Clinical Prediction Models: An Illustrative Case Study.

The performance of clinical prediction models tends to deteriorate over time. Researchers often develop a new prediction if an existing model performs poorly at external validation. Model updating is an efficient technique and promising alternative to the de novo development of clinical prediction models. Model updating has been recommended by the TRIPOD guidelines. To illustrate several model updating techniques, a case study is provided for the development and updating of a clinical prediction model assessing postoperative anxiety in data coming from two double-blinded placebo-controlled randomized controlled trials with a very similar methodological framework. Note that the developed model and updated model are for didactic purposes only. This paper discusses some common considerations and caveats for researchers to be aware of when planning or applying updating of a prediction model.

ROC curves for clinical prediction models part 1. ROC plots showed no added value above the AUC when evaluating the performance of clinical prediction models

ObjectivesReceiver operating characteristic (ROC) curves show how well a risk prediction model discriminates between patients with and without a condition. We aim to investigate how ROC curves are presented in the literature and discuss and illustrate their potential limitations. Study Design and SettingWe conducted a pragmatic literature review of contemporary publications that externally validated clinical prediction models. We illustrated limitations of ROC curves using a testicular cancer case study and simulated data. ResultsOf 86 identified prediction modeling studies, 52 (60%) presented ROC curves without thresholds and one (1%) presented an ROC curve with only a few thresholds. We illustrate that ROC curves in their standard form withhold threshold information have an unstable shape even for the same area under the curve (AUC) and are problematic for comparing model performance conditional on threshold. We compare ROC curves with classification plots, which show sensitivity and specificity conditional on risk thresholds. ConclusionROC curves do not offer more information than the AUC to indicate discriminative ability. To assess the model's performance for decision-making, results should be provided conditional on risk thresholds. Therefore, if discriminatory ability must be visualized, classification plots are attractive.

ROC curves for clinical prediction models part 3. The ROC plot: a picture that needs a 1000 words

In our recent perspective, we argue that receiver operating characteristic (ROC) curves add no useful information to the area under the ROC curve (AUC) for the evaluation of discriminatory ability of risk prediction models [ [1] Verbakel J.Y. Steyerberg E.W. Uno H. De Cock B. Wynants L. Collins G.S. et al. ROC plots showed no added value above the AUC when evaluating the performance of clinical prediction models. J Clin Epidemiol. 2020; Abstract Full Text Full Text PDF PubMed Scopus (9) Google Scholar ]. Our key argument is that a risk threshold needs to be considered if a model is used to support decision-making, while ROC curves in their standard form suppress threshold information. Despite her critical assessment of our article, Janssens raised no strong or convincing arguments against this view [ [2] Janssens A.C.J.W. The ROC plot: the picture that could be worth a 1000 words. J Clin Epidemiol. 2020; Abstract Full Text Full Text PDF Scopus (2) Google Scholar ]. Janssens states that the ROC curve is valid, and that it is subjective whether it has added value over the AUC alone: “others may benefit from seeing how and how much (little) the addition of predictors improves the discriminative ability”. However, that is exactly what the AUC quantifies, and Janssens seemingly agrees: “the AUC and ROC plot present the overall discriminative ability of prediction models”.

Impact of predictor measurement heterogeneity across settings on the performance of prediction models: A measurement error perspective.

It is widely acknowledged that the predictive performance of clinical prediction models should be studied in patients that were not part of the data in which the model was derived. Out‐of‐sample performance can be hampered when predictors are measured differently at derivation and external validation. This may occur, for instance, when predictors are measured using different measurement protocols or when tests are produced by different manufacturers. Although such heterogeneity in predictor measurement between derivation and validation data is common, the impact on the out‐of‐sample performance is not well studied. Using analytical and simulation approaches, we examined out‐of‐sample performance of prediction models under various scenarios of heterogeneous predictor measurement. These scenarios were defined and clarified using an established taxonomy of measurement error models. The results of our simulations indicate that predictor measurement heterogeneity can induce miscalibration of prediction and affects discrimination and overall predictive accuracy, to extents that the prediction model may no longer be considered clinically useful. The measurement error taxonomy was found to be helpful in identifying and predicting effects of heterogeneous predictor measurements between settings of prediction model derivation and validation. Our work indicates that homogeneity of measurement strategies across settings is of paramount importance in prediction research.

Poor performance of clinical prediction models: the harm of commonly applied methods

ObjectiveTo evaluate limitations of common statistical modeling approaches in deriving clinical prediction models and explore alternative strategies. Study Design and SettingA previously published model predicted the likelihood of having a mutation in germline DNA mismatch repair genes at the time of diagnosis of colorectal cancer. This model was based on a cohort where 38 mutations were found among 870 participants, with validation in an independent cohort with 35 mutations. The modeling strategy included stepwise selection of predictors from a pool of over 37 candidate predictors and dichotomization of continuous predictors. We simulated this strategy in small subsets of a large contemporary cohort (2,051 mutations among 19,866 participants) and made comparisons to other modeling approaches. All models were evaluated according to bias and discriminative ability (concordance index, c) in independent data. ResultsWe found over 50% bias for five of six originally selected predictors, unstable model specification, and poor performance at validation (median c = 0.74). A small validation sample hampered stable assessment of performance. Model prespecification based on external knowledge and using continuous predictors led to better performance (c = 0.836 and c = 0.852 with 38 and 2,051 events respectively). ConclusionPrediction models perform poorly if based on small numbers of events and developed with common but suboptimal statistical approaches. Alternative modeling strategies to best exploit available predictive information need wider implementation, with collaborative research to increase sample sizes.

A tutorial on variable selection for clinical prediction models: feature selection methods in data mining could improve the results

ObjectivesIdentifying an appropriate set of predictors for the outcome of interest is a major challenge in clinical prediction research. The aim of this study was to show the application of some variable selection methods, usually used in data mining, for an epidemiological study. We introduce here a systematic approach. Study Design and SettingThe P-value-based method, usually used in epidemiological studies, and several filter and wrapper methods were implemented to select the predictors of diabetes among 55 variables in 803 prediabetic females, aged ≥20 years, followed for 10–12 years. To develop a logistic model, variables were selected from a train data set and evaluated on the test data set. The measures of Akaike information criterion (AIC) and area under the curve (AUC) were used as performance criteria. We also implemented a full model with all 55 variables. ResultsWe found that the worst and the best models were the full model and models based on the wrappers, respectively. Among filter methods, symmetrical uncertainty gave both the best AUC and AIC. ConclusionOur experiment showed that the variable selection methods used in data mining could improve the performance of clinical prediction models. An R program was developed to make these methods more feasible and visualize the results.