AUC Measures Research Articles

Clinical Utility in Evaluation of Risk Models

Statistical and clinical perspectives on risk models can be very different, even with agreement on the objective: to develop accurate and precise risk estimates for rational, effective, and cost-effective prevention strategies (1). For example, Katki et al. (2) estimated the risk of cervical precancer (cervical intraepithelial neoplasia grade 3) or worse conditional on results of concurrent testing for human papillomavirus (HPV) and cervical cytology (cotesting) in 300,000 women 30 years of age or older who were enrolled in a large health maintenance organization. The nearly equivalent risk over 5 years for all the women who were HPV negative (3.8 per 100,000 per year) and those who were both HPV negative and cytology negative (3.2 per 100,000 per year) might justify a prevention program using primary screening based on results of HPV testing alone. In contrast, among 17,000 HPV-positive women, the 5-year risk of cervical intraepithelial neoplasia grade 3 or worse was substantially lower (5.9%) in the 12,000 women with normal cytology than it was in the 5,000 women who had abnormal cytology (12.1%). This suggests that cytology testing might improve decisions about management of women who test positive for HPV. As exemplified by Katki et al. (2), the value of an additional clinical test is a function of simple risks or generalizations of predictive values, along with proportions with the different clinical presentations. Both a risk model and a clinical test are risk classifiers or risk stratifiers. For evaluation of risk models, as Cook (3) notes in one commentary discussing an article by Pencina et al. (4) in this issue of the Journal, neither area under the receiver operating characteristic curve (AUC), integrated discrimination improvement (IDI), nor net reclassification improvement (NRI) is necessary when direct evaluation of clinical performance of a program based on the risk model is possible. The measure of clinical utility is difference in rates or risk of clinical outcomes from intervention strategies using rules for assignment based on the different models. The differences in risk and the distribution of the possible clinical presentation from the models are measures of risk stratification or of the extra precision resulting from the added complexity of a model. Of course, clinical utility also depends on the efficacy of available interventions; a model with prefect predictions has no clinical utility without an effective intervention. Pencina et al. (4) discussed measures for evaluating risk models by purely statistical assessment without consideration of interventions with known efficacy, which is needed for evaluating clinical utility. They advocated reporting IDI and NRI in addition to the most commonly used measure, AUC. AUC is a nonparametric test statistic (the Mann–Whitney U test equivalent to a Wilcoxon rank sum test) of equality of distribution of estimated risk in cases and controls. The IDI adds to the AUC because an estimate of difference in means adds information not available from a statistic that is based on ranks and not actual values. NRIs can capture effects of differences in distribution, including spread and skewness, that are not reflected in the difference in means. As noted by Pencina et al. (4), 2 of the American Heart Association's 2009 criteria for evaluation of novel markers of cardiovascular risk are “documentation of incremental information when added to standard risk markers” and “assessment of effects on patient management and outcomes” (5, p. 2408). As noted by Kerr et al. (6), assessment of the marginal increase in utility from an improved model requires an objective assessment of effects on patient management and outcomes; the incremental information criterion needs to be a purely statistical measure if it is separate from clinical utility. The difference between the clinical and statistical views is manifested in one particularly important way: the role of variability in the risk. Less variability, or more homogeneity, in risk within cases and controls increases the AUC and other measures of discrimination; in contrast, greater variation in risk in the study population for which decisions are to be made increases the potential for assigning a intervention for those at extreme risks different from the intervention appropriate for one at average risk. In other words, small variance in risk, conditional on disease, increases discrimination, but large unconditional variance increases the potential for clinical utility. Of course, the variation needs to be real, not a consequence of random variation of risk estimates or misclassification of markers in the model. This exchange of views (1, 2, 6, 7) highlights the differences between the clinical and statistical perspectives on risk models. The connection between the 2 perspectives is not yet clear to all.

Predicting the need for CT imaging in children with minor head injury using an ensemble of Naive Bayes classifiers

Using an automatic data-driven approach, this paper develops a prediction model that achieves more balanced performance (in terms of sensitivity and specificity) than the Canadian Assessment of Tomography for Childhood Head Injury (CATCH) rule, when predicting the need for computed tomography (CT) imaging of children after a minor head injury. CT is widely considered an effective tool for evaluating patients with minor head trauma who have potentially suffered serious intracranial injury. However, its use poses possible harmful effects, particularly for children, due to exposure to radiation. Safety concerns, along with issues of cost and practice variability, have led to calls for the development of effective methods to decide when CT imaging is needed. Clinical decision rules represent such methods and are normally derived from the analysis of large prospectively collected patient data sets. The CATCH rule was created by a group of Canadian pediatric emergency physicians to support the decision of referring children with minor head injury to CT imaging. The goal of the CATCH rule was to maximize the sensitivity of predictions of potential intracranial lesion while keeping specificity at a reasonable level. After extensive analysis of the CATCH data set, characterized by severe class imbalance, and after a thorough evaluation of several data mining methods, we derived an ensemble of multiple Naive Bayes classifiers as the prediction model for CT imaging decisions. In the first phase of the experiment we compared the proposed ensemble model to other ensemble models employing rule-, tree- and instance-based member classifiers. Our prediction model demonstrated the best performance in terms of AUC, G-mean and sensitivity measures. In the second phase, using a bootstrapping experiment similar to that reported by the CATCH investigators, we showed that the proposed ensemble model achieved a more balanced predictive performance than the CATCH rule with an average sensitivity of 82.8% and an average specificity of 74.4% (vs. 98.1% and 50.0% for the CATCH rule respectively). Automatically derived prediction models cannot replace a physician's acumen. However, they help establish reference performance indicators for the purpose of developing clinical decision rules so the trade-off between prediction sensitivity and specificity is better understood.

Insights into the area under the receiver operating characteristic curve (AUC) as a discrimination measure in species distribution modelling

ABSTRACTAim The area under the receiver operating characteristic (ROC) curve (AUC) is a widely used statistic for assessing the discriminatory capacity of species distribution models. Here, I used simulated data to examine the interdependence of the AUC and classical discrimination measures (sensitivity and specificity) derived for the application of a threshold. I shall further exemplify with simulated data the implications of using the AUC to evaluate potential versus realized distribution models.Innovation After applying the threshold that makes sensitivity and specificity equal, a strong relationship between the AUC and these two measures was found. This result is corroborated with real data. On the other hand, the AUC penalizes the models that estimate potential distributions (the regions where the species could survive and reproduce due to the existence of suitable environmental conditions), and favours those that estimate realized distributions (the regions where the species actually lives).Main conclusions Firstly, the independence of the AUC from the threshold selection may be irrelevant in practice. This result also emphasizes the fact that the AUC assumes nothing about the relative costs of errors of omission and commission. However, in most real situations this premise may not be optimal. Measures derived from a contingency table for different cost ratio scenarios, together with the ROC curve, may be more informative than reporting just a single AUC value. Secondly, the AUC is only truly informative when there are true instances of absence available and the objective is the estimation of the realized distribution. When the potential distribution is the goal of the research, the AUC is not an appropriate performance measure because the weight of commission errors is much lower than that of omission errors.

Using species distribution models to identify suitable areas for biofuel feedstock production

AbstractThe 2007 Energy Independence and Security Act mandates a five‐fold increase in US biofuel production by 2022. Given this ambitious policy target, there is a need for spatially explicit estimates of landscape suitability for growing biofuel feedstocks. We developed a suitability modeling approach for two major US biofuel crops, corn (Zea mays) and switchgrass (Panicum virgatum), based upon the use of two presence‐only species distribution models (SDMs): maximum entropy (Maxent) and support vector machines (SVM). SDMs are commonly used for modeling animal and plant distributions in natural environments, but have rarely been used to develop landscape models for cultivated crops. AUC, Kappa, and correlation measures derived from test data indicate that SVM slightly outperformed Maxent in modeling US corn production, although both models produced significantly accurate results. When compared with results from a mechanistic switchgrass model recently developed by Oak Ridge National Laboratory (ORNL), SVM results showed higher correlation than Maxent results with models fit using county‐scale point inputs of switchgrass production derived from expert opinion estimates. However, Maxent results for an alternative switchgrass model developed with point inputs from research trial sites showed higher correlation to the ORNL model than the corresponding results obtained from SVM. Further analysis indicates that both modeling approaches were effective in predicting county‐scale increases in corn production from 2006 to 2007, a time period in which US corn production increased by 24%. We conclude that presence‐only methods are a powerful first‐cut tool for estimating relative land suitability across geographic regions in which candidate biofuel feedstocks can be grown, and may also provide important insight into potential land‐use change patterns likely to be associated with increased biofuel demand.

CONTINUOUS MEASURES OF BREATHING AND PHASE ANGLE IN CONSCIOUS CYNOMOLGUS MONKEYS

We evaluated an inductance plethysmography (IP) system in 3 allergic cynomolgus monkeys for measurement of breathing and phase angle (Φ) as a surrogate for airway resistance. Initial studies in anesthetized, intubated animals compared a standard pneumotachograph measure of ventilation with IP to estimate the accuracy of the measurement. Data showed excellent agreement between IP and the standard approach. 24 hours of baseline unanesthetized breathing was then recorded. After 24 hours, monkeys were re‐anesthetized, exposed to an aerosolized ascaris suum antigen and immediately returned to their cages. A subsequent 24 hours of recording provided data from each animal during and following a bronchoconstrictor response. Data indicate that ventilatory pattern is distinctly altered for a period of at least 60 minutes post challenge at which point breathing recovers to normal values. These data agree with our AUC measures in anesthetized animals after antigen exposure. The Φ for each breath was also calculated. During bronchoconstriction, a distinct different angle was measurable. Upon recovery, Φ returned to normal and remained stable, regardless breathing pattern. We conclude that this system is a useful tool to measure breathing in unencumbered, conscious animals for a long period and may be useful in evaluating bronchoconstrictor events.

Effects of nateglinide on the secretion of glycated insulin and glucose tolerance in type 2 diabetes

Aims: Glycation of insulin has been demonstrated within pancreatic β-cells and the resulting impaired bioactivity may contribute to insulin resistance in diabetes. We used a novel radioimmunoassay to evaluate the effect of nateglinide on plasma concentrations of glycated insulin and glucose tolerance in type 2 diabetes. Methods: Ten patients (5 M/5 F, age 57.8±1.9 years, HbA 1c 7.6±0.5%, fasting plasma glucose 9.4±1.2 mmol/l, creatinine 81.6±4.5 μmol/l) received oral nateglinide 120 mg or placebo, 10 min prior to 75 g oral glucose in a random, single blind, crossover design, 1 week apart. Blood samples were taken for glycated insulin, glucose, insulin and C-peptide over 225 min. Results: Plasma glucose and glycated insulin responses were reduced by 9% ( P=0.005) and 38% ( P=0.047), respectively, following nateglinide compared with placebo. Corresponding AUC measures for insulin and C-peptide were enhanced by 36% ( P=0.005) and 25% ( P=0.007) by nateglinide. Conclusions: Glycated insulin in type 2 diabetes is reduced in response to the insulin secretagogue nateglinide, resulting in preferential release of native insulin. Since glycated insulin exhibits impaired biological activity, reduced glycated insulin release may contribute to the antihyperglycaemic action of nateglinide.

The trigonometric responder approach: a new method for detecting responders to pharmacological or experimental challenges.

The paper presents a newly developed response measure that is particularly suitable for the evaluation of pharmacokinetic data. This method is based on trigonometric considerations, defining a hormone response as the difference between the angle of the slope of the curve before and after drug intake. In addition, the size of this difference is compared to the difference obtained in placebo conditions. In this way, the trigonometric response measure overcomes one of the most problematic shortcomings of the 'area under the curve' (AUC) approach, the problem of the initial value. We will present the mathematical background of the trigonometric method and demonstrate its usefulness by evaluating empirical data (a pharmacological challenge test using the dopamine agonist lisuride) and comparing it to classical AUC measures. This has been achieved by contrasting both approaches with responder definitions according to binary time series analysis and the peak value of the curve.