Re: "Clinical Usefulness of the Framingham Cardiovascular Risk Profile beyond Its Statistical Performance: The Tehran Lipid and Glucose Study"

Abstract

Khalili et al. (1) evaluated the clinical utility of the Framingham cardiovascular risk model in a Middle Eastern cohort study. We commend the authors for their use of highly relevant measures of model performance, the net benefit (2, 3) and net benefit fraction (4, 5). However, we would like to make several points regarding their interpretation and reporting of these measures. Interpretation of the net benefit and net benefit fraction The population net benefit (NB) of a risk model, used to classify individuals as being at high 10-year risk of cardiovascular disease using the risk threshold rH, is (1) where B is the benefit of classifying as high-risk a subject who would develop cardiovascular disease absent treatment (a “cardiovascular event”), H is the harm of classifying as high-risk a subject who would not develop cardiovascular disease absent treatment (a “cardiovascular nonevent”), TPR(rH) and FPR(rH) are the true- and false-positive rates associated with this risk threshold, and ρ is the proportion of subjects who experience cardiovascular events. Vickers and Elkin (2) set B to 1, so that the units are true positive classifications, and use the optimal risk threshold rH = H/H + B (6) in order to rewrite equation 1 as (2) Baker et al. (4, 5) scale equation 2 by ρ, which is the maximum possible net benefit for a model that classifies as high-risk all cardiovascular events and all others as low-risk. The net benefit fraction (NBF) is therefore (3) Khalili et al. interpret NBF(0.2) = 0.156 for the Framingham risk model applied to women as “the fraction of the incidence rate that could be predicted and prevented” (1, p. 178). However, this is not a valid interpretation. Their interpretation seems to relate to the TPR of the model, whereas NBF(0.2) = 0.156 is more accurately interpreted as a discounted TPR; the TPR is offset by the FPR where false-positives are put on the same scale as true-positives (7). We prefer to interpret NBF(0.2) = 0.156 as the same benefit that would be achieved by classifying as high-risk 15.6% of cardiovascular events and none of the nonevents (2, 3). The authors' interpretation also suggests that all cardiovascular events classified as high-risk would necessarily be prevented; however, this would depend on the efficacy of subsequent interventions. Net benefit for different subsets of the population Net benefit curves, which plot the net benefit in equation 2 versus the risk threshold rH (2, 3), are also used by Khalili et al. The authors recommend these curves for examining the net benefit for a range of risk thresholds, given that different patient subgroups may have different side effects of interventions and therefore different risk thresholds. However, the net benefit curve shows the total population net benefit at each risk threshold. Two points on the curve cannot be interpreted as the net benefits for 2 different subpopulations because, if the risk distribution in the subpopulation differs from the risk distribution in the whole population, the net benefit in the subpopulation is different from the net benefit in the whole population at that threshold. For example, men and women in the Middle Eastern cohort have different values for TPR, FPR, and ρ and therefore have net benefits that differ from the population net benefit shown on the net benefit curve. Recommendation We have several practical recommendations. The first is to stratify any analysis by important patient characteristics that affect risk threshold—for example, gender as in the study by Khalili et al. (1). The second is to report the net benefit fraction together with its constituents, the TPR and FPR, over a range of risk thresholds. Seeing these individual components helps in digesting the net benefit. Knowing the TPR and FPR is also helpful when risk thresholds are chosen “irrationally.” That is, the net benefit in equation 2 assumes that the risk threshold reflects the cost-benefit ratio, rH = H/H + B. However, individuals or policy-makers may choose risk thresholds in ways that do not reflect the cost-benefit ratio. For example, in their commentary accompanying Khalili et al.'s article, D'Agostino and Pencina (8) suggest that the risk threshold rH = 0.2 was chosen without regard to cost-benefit considerations. Reporting the TPR and FPR as a function of risk threshold allows the reader to choose a threshold and to perform any calculus desired to take into consideration both components of model performance.