What Makes a Good Predictor?

Abstract

EACH YEAR, RESEARCHERS IDENTIFY THOUSANDS OF potential new “tools” for predicting patients’ medical futures. There is heightened interest for discovering, validating, and incorporating into clinical practice predictors that improve treatment choices and outcomes thereof. Thousands of articles report on potential predictors. A search of PubMed clinical queries under prognosis (specific strategy) yields 165 746 articles for cancer, 72 354 for cardiovascular disease, and even 3749 for rheumatoid arthritis. These run the gamut, including genetic tests, biomarkers, and an increasing variety of imaging modes, lengthening the list of candidate predictors. However, very few of these proposed predictors eventually change practice. Why? What makes a good predictor? A good predictor is one that has a favorable risk-benefit ratio, reasonable cost, acceptability, and convenience. As for any intervention in health care, proper evidence ideally requires randomized trials demonstrating that using the predictor improves decision making and subsequent clinical outcomes without inordinate adverse events. It also requires formal cost-effectiveness analyses, integrating benefits, risks, and cost. However, hardly any of the predictors in the literature or even those routinely adopted in clinical practice have had their effectiveness proven in randomized trials. Only a few examples of such trials exist; eg, trials evaluating the benefits of screening for abdominal aneurysms or measuring brain-type natriuretic peptide in patients with dyspnea. Conversely, a comprehensive randomized trial agenda trying to evaluate every proposed predictor in each proposed disease application and population would require millions of trials, which is unrealistic. Which candidate predictors should be evaluated by randomized trials and how should they be chosen for best results? A commonsense checklist might be to, first, preferably test predictors for diseases with major morbidity. Second, some effective treatment should be available. Third, the treatment should not be equally effective (or equally risky) for all persons. Fourth, consideration of the predictor should allow more accurate classification of individuals into categories in which treatment is or is not indicated. Fifth, the incremental prediction should be accomplished beyond what can be achieved with information already available. Sixth, there should be consensus about and standardization of established, routine predictors. Seventh, the predictor should be unambiguously defined and measured. Most published research on predictors is irrelevant or tangential to this checklist. Almost all articles report statistically significant results, but this means little. Many investigators deal with whether a predictor in isolation has any ability to predict something. This, however, does not consider that many clinical facts and routine laboratory predictors may already inform prognosis. Thus, it is often not clear whether the new test adds incremental prognostic information beyond known factors. Much of the literature is chaotic, and data dredging and selective reporting abound. Strong studies with clear design, purpose, and knowledge are clearly needed. In this issue of JAMA, Polonsky et al present such a welldesigned study addressing coronary artery calcium score (CACS) as a predictor of coronary heart disease (CHD). Is this predictor good enough? In regard to the aforementioned checklist, first, CHD indeed carries major morbidity. Second, effective lipid-lowering treatments are available for preventive purposes. Third, the absolute effectiveness of the treatments (absolute risk reduction) varies at different categories of baseline risk. Patients at greater than 20% risk of CHD over 10 years should be treated, those with less than 10% should not, and those with 10% to 20% are in the gray zone of intermediate risk. Fourth, Polonsky et al suggest that CACS does allow for a better classification of patients into categories in which, seemingly, treatment is or not indicated. Fifth, this is accomplished in addition to the information available from established routine predictors,