Re: Combined associations of genetic and environmental risk factors: implications for prevention of breast cancer.

Abstract

In a recent commentary, Garcia-Closas and colleagues (1) discussed two metrics to evaluate the addition of genetic information to established risk factors in models to predict the risk of invasive breast cancer in asymptomatic women: 1) change in the area under the receiver operating characteristic curve (AUC) and 2) change in the population distribution of relative risk (RR) of invasive breast cancer, where RR is defined with respect to average risk. Comparing Model 1, with established risk factors, and Model 7, which added a polygenic risk score to established risk factors, Garcia-Closas and colleagues (1) found that the AUC increased from 0.618 to 0.703 and the percent in the population (percent in cases) with RR between 2 and 3 increased from 3.8% (8.8%) to 6.0% (14.4%) and the percent with RR greater than 3 increased from 1.0% (3.7%) to 3.8% (16.0%). Garcia-Closas and colleagues (1) opined that AUC is not a good metric because it “cannot capture the degree of stratification of absolute risk that a new risk factor can add to the model” and concluded that the change in RR distribution indicated “substantial improvement in risk stratification.” However, there is a fundamental limitation that applies to both of the aforementioned metrics; namely, they do not incorporate harms and benefits associated with clinical decision-making. Without incorporating harms and benefits, it is not possible to choose between different statistical methods for evaluating risk prediction (2). Approaches that account for costs and benefit include decision curves (3) and relative utility curves (2,4,5). These approaches involve a sensitivity analysis for the risk threshold, the risk at which persons are indifferent between treatment (here chemoprevention) and no treatment (no chemoprevention). A full decision curve or relative utility curve analysis requires data not available in the article. Nevertheless the following back-of-the-envelope calculation provides perspective. Suppose the receiver operating characteristic (ROC) curve is based on a constant odds ratio (OR) model, namely OR = {true-positive rate [TPR] (1 – false positive rate [FPR])} / {(1–TPR) FPR}, as shown in Figure 1. For this ROC curve, AUC = OR(OR – 1– log[OR]) / (OR– 1)2. Relative utility is the utility of risk prediction divided by the utility of perfect prediction evaluated at a risk threshold. As derived in the Appendix of Baker et al. (5), for this constant odds ratio ROC curve, the maximum relative utility (maxRU) is (√OR– 1) / (√OR+ 1), which, incidentally, equals Yule’s 1912 coefficient of colligation (6), although they have very different rationales. For an OR = 2, the AUC = 0.62 and maxRU = 0.17. For an OR = 4, the AUC = 0.72 and maxRU = 0.33. If the probability of breast cancer in the target population is 0.001, the test tradeoff (5) comparing models with OR = 2 vs OR = 4 equals 1/{(0.33 – 0.17) .001} = 6250. This test tradeoff implies that genetic information needs to be collected on at least 6250 persons for every correct prediction of invasive breast in order to increase the net benefit. Figure 1. Receiver operating characteristic (ROC) and relative utility curves for the constant odds ratio ROC curve. FPR = false-positive rate; OR = odds ratio; ROC = receiver operating characteristic curve; TPR = true-positive rate.