Discriminative Ability Of Prediction Model Research Articles

Propensity-based standardization to enhance the validation and interpretation of prediction model discrimination for a target population.

External validation of the discriminative ability of prediction models is of key importance. However, the interpretation of such evaluations is challenging, as the ability to discriminate depends on both the sample characteristics (ie, case-mix) and the generalizability of predictor coefficients, but most discrimination indices do not provide any insight into their respective contributions. To disentangle differences in discriminative ability across external validation samples due to a lack of model generalizability from differences in sample characteristics, we propose propensity-weighted measures of discrimination. These weighted metrics, which are derived from propensity scores for sample membership, are standardized for case-mix differences between the model development and validation samples, allowing for a fair comparison of discriminative ability in terms of model characteristics in a target population of interest. We illustrate our methods with the validation of eight prediction models for deep vein thrombosis in 12 external validation data sets and assess our methods in a simulation study. In the illustrative example, propensity score standardization reduced between-study heterogeneity of discrimination, indicating that between-study variability was partially attributable to case-mix. The simulation study showed that only flexible propensity-score methods (allowing for non-linear effects) produced unbiased estimates of model discrimination in the target population, and only when the positivity assumption was met. Propensity score-based standardization may facilitate the interpretation of (heterogeneity in) discriminative ability of a prediction model as observed across multiple studies, and may guide model updating strategies for a particular target population. Careful propensity score modeling with attention for non-linear relations is recommended.

Assessing the probability of risk factor control in patients with coronary heart disease: results from the ESC-EORP EUROASPIRE V survey.

In patients with coronary heart disease (CHD), we investigated whether it is possible to accurately assess the probability of short-term control of risk factors (blood pressure, cholesterol, smoking) based on individual and large-area residential characteristics. We merged individual data of participants from EUROASPIRE V who were hospitalized for CHD (2014-2017) and interviewed and examined for risk factor control (2016-2017), with large-area residential data provided by Eurostat for Nomenclature of Territorial Units for Statistics (NUTS) regions using postal codes. Data from 2562 CHD patients in 16 countries were linked to data from 60 NUTS 2 and 121 NUTS 3 regions. The median time between hospitalization and interview was 14 months. We developed prediction models to assess the probability of risk factor control at interview using data from the time of hospitalization: (i) baseline models including 35 variables on patients' demographic, clinical, and socio-economic characteristics and (ii) extended models additionally considering nine variables on large-area residential characteristics. We calculated and internally validated c-indices to assess the discriminative ability of prediction models. Baseline models showed good discrimination with c-indices of 0.69, 0.70, and 0.76 for blood pressure control, cholesterol control, and smoking cessation, respectively. Extended models for blood pressure, cholesterol, and smoking yielded improved c-indices of 0.72, 0.71, and 0.78, respectively. Our results indicate that the probability of risk factor control in CHD patients can be accurately assessed using individual and large-area residential characteristics, allowing for an identification of patients who are less likely to achieve risk factor targets.

Predicting risk factor control in patients with coronary heart disease using patients' individual and large-area residential characteristics: results from the ESC-EORP EUROASPIRE V survey

Abstract Introduction The impact of small-area residential characteristics on the development of coronary heart disease (CHD) and cardiovascular risk factors is well established. Evidence is so far lacking on the predictive value of large-area residential characteristics for cardiovascular risk. Purpose In CHD patients, we aimed to study the predictive value of large-area residential characteristics beyond patients' individual characteristics for the control of major cardiovascular risk factors (blood pressure, cholesterol and smoking). Methods Patients hospitalised for coronary events or interventions from 16 European countries were interviewed and examined for risk factor control (EUROASPIRE V survey). Based on patients' postal codes, we merged individual patient data with large-area residential data routinely provided by Eurostat for NUTS regions (Nomenclature of Territorial Units for Statistics). We selected patient characteristics recorded at hospitalisation (2015–16) and socioeconomic characteristics of their residential NUTS 2 and 3 regions (2015–16) to predict risk factor control at interview (2016–17). We developed risk prediction models using multi-level logistic regression: 1) baseline models (BMs) with patient data only and 2) extended models (EMs) with patient data (level 1) and large-area residential data (level 2). We employed multiple imputation to overcome sparse data and internally validated results using cross-validation. We used the c-index corresponding to the area under the curve as performance measure to assess the discriminative ability of prediction models. Results Data from 2562 CHD patients in 16 countries could be linked to 60 NUTS 2 and 121 NUTS 3 regions by postal codes. Median time between hospitalisation and interview was 14 (range 6 to 28) months. BMs included 34 patient variables, covering demographic, socioeconomic and clinical characteristics, and EMs additionally included 11 regional socioeconomic variables concerning gross domestic product, income, education, occupation, population density, and health care. For blood pressure control, BMs and EMs showed validated c-indices ranging from 0.71 to 0.73 and from 0.73 to 0.77, respectively. Analyses for cholesterol control yielded c-indices ranging from 0.69 to 0.70 in BMs and from 0.71 to 0.73 in EMs. For smoking cessation, the c-indices ranged from 0.80 to 0.84 in BMs and from 0.83 to 0.84 in EMs. Conclusions Prediction models based on CHD patients' individual characteristics showed a high discriminative ability regarding the control of major cardiovascular risk factors. Further consideration of large-area residential characteristics provided an additive predictive value, markedly increasing the discriminative ability of prediction models for blood pressure and cholesterol control. Socioeconomic characteristics of CHD patients' residential regions can thus help identify patients requiring more intense risk factor management. Funding Acknowledgement Type of funding source: Foundation. Main funding source(s): German Heart Foundation

ROC curves for clinical prediction models part 3. The ROC plot: a picture that needs a 1000 words

In our recent perspective, we argue that receiver operating characteristic (ROC) curves add no useful information to the area under the ROC curve (AUC) for the evaluation of discriminatory ability of risk prediction models [ [1] Verbakel J.Y. Steyerberg E.W. Uno H. De Cock B. Wynants L. Collins G.S. et al. ROC plots showed no added value above the AUC when evaluating the performance of clinical prediction models. J Clin Epidemiol. 2020; Abstract Full Text Full Text PDF PubMed Scopus (9) Google Scholar ]. Our key argument is that a risk threshold needs to be considered if a model is used to support decision-making, while ROC curves in their standard form suppress threshold information. Despite her critical assessment of our article, Janssens raised no strong or convincing arguments against this view [ [2] Janssens A.C.J.W. The ROC plot: the picture that could be worth a 1000 words. J Clin Epidemiol. 2020; Abstract Full Text Full Text PDF Scopus (2) Google Scholar ]. Janssens states that the ROC curve is valid, and that it is subjective whether it has added value over the AUC alone: “others may benefit from seeing how and how much (little) the addition of predictors improves the discriminative ability”. However, that is exactly what the AUC quantifies, and Janssens seemingly agrees: “the AUC and ROC plot present the overall discriminative ability of prediction models”.

Reflection on modern methods: Revisiting the area under the ROC Curve.

The area under the receiver operating characteristic (ROC) curve (AUC) is commonly used for assessing the discriminative ability of prediction models even though the measure is criticized for being clinically irrelevant and lacking an intuitive interpretation. Every tutorial explains how the coordinates of the ROC curve are obtained from the risk distributions of diseased and non-diseased individuals, but it has not become common sense that therewith the ROC plot is just another way of presenting these risk distributions. We show how the ROC curve is an alternative way to present risk distributions of diseased and non-diseased individuals and how the shape of the ROC curve informs about the overlap of the risk distributions. For example, ROC curves are rounded when the prediction model included variables with similar effect on disease risk and have an angle when, for example, one binary risk factor has a stronger effect; and ROC curves are stepped rather than smooth when the sample size or incidence is low, when the prediction model is based on a relatively small set of categorical predictors. This alternative perspective on the ROC plot invalidates most purported limitations of the AUC and attributes others to the underlying risk distributions. AUC is a measure of the discriminative ability of prediction models. The assessment of prediction models should be supplemented with other metrics to assess their clinical utility.

178 Prediction of Optimal Intervals of Radiological Surveillance for Workers at Different Risks of Silicosis - China’s Experience

ObjectivesTo determine the optimal intervals of chest radiographic surveillance for workers at different risks of silicosis.MethodsAll 3492 workers who were exposed to silica dust during 1964–74 in an iron-ore of...