How well do prediction equations predict? Using receiver operating characteristic curves and accuracy curves to compare validity and generalizability.

Abstract

Although morbidity and mortality prediction equations are widely used in planning, clinical practice, and health risk appraisal, their validity and generalizability have been tested only in a limited way. Previous attempts lacked an absolute standard of performance and looked only at the equations' ability to predict who would become ill (sensitivity), not the equally important ability to predict who would remain healthy (specificity). We compared six all-cause mortality prediction equations using receiver operating characteristic curves and accuracy curves, which overcome the limitations of earlier methods and provide a concise visual representation of the results. We used equations from five prospective studies conducted in the United States (Tecumseh at 8 and 12 years of follow-up, Framingham, Chicago Gas, Chicago Western Electric, and Albany), each of which included cholesterol, smoking, and blood pressure as independent variables, to predict 12-year mortality in Tecumseh males age 40-54 years. Previous studies suggested that these equations predict equally well. Our analysis found that, although all predict better than chance, Albany, Chicago Western Electric, and Tecumseh at 8 years underestimate mortality. Receiver operating characteristic and accuracy curves are a promising technique for assessment of prediction equations.