Assessment of Improved Prediction Beyond Traditional Risk Factors

Abstract

HomeCirculation: Cardiovascular GeneticsVol. 3, No. 1Assessment of Improved Prediction Beyond Traditional Risk Factors Free AccessResearch ArticlePDF/EPUBAboutView PDFSections ToolsAdd to favoritesDownload citationsTrack citationsPermissions ShareShare onFacebookTwitterLinked InMendeleyReddit Jump toFree AccessResearch ArticlePDF/EPUBAssessment of Improved Prediction Beyond Traditional Risk FactorsWhen Does a Difference Make a Difference? A. Cecile J.W. Janssens, PhD and Muin J. Khoury, MD, PhD A. Cecile J.W. JanssensA. Cecile J.W. Janssens From the Department of Epidemiology (A.C.J.W.J.), Erasmus University Medical Center, Rotterdam, The Netherlands; The Office of Public Health Genomics, Centers for Disease Control and Prevention (M.J.K.), Atlanta, Ga. Search for more papers by this author and Muin J. KhouryMuin J. Khoury From the Department of Epidemiology (A.C.J.W.J.), Erasmus University Medical Center, Rotterdam, The Netherlands; The Office of Public Health Genomics, Centers for Disease Control and Prevention (M.J.K.), Atlanta, Ga. Search for more papers by this author Originally published1 Feb 2010https://doi.org/10.1161/CIRCGENETICS.110.938092Circulation: Cardiovascular Genetics. 2010;3:3–5The moderate predictive performance of cardiovascular disease risk models necessitates more studies that investigate the incremental value of novel biomarkers. In recent years, many new biomarkers have been evaluated for their ability to improve prediction of cardiovascular disease beyond traditional risk factors, including C-reactive protein, coronary artery calcium, and single-nucleotide polymorphisms in the 9p21 region.1 The interest in novel biomarkers is propelled, in part, by emerging discoveries from genomewide association studies of genetic variants associated with risk for many common diseases.2 Nonetheless, family history, an “old” tool in clinical practice—crucial for the diagnosis and management of genetic disorders—has not been adequately explored for its value in risk assessment and prevention of common diseases.3 Positive family history is a strong risk factor for cardiovascular diseases,4,5 reflecting the consequences of genetic and nongenetic risk factors that are shared among relatives. From a practical perspective, family history is a strong determinant of disease that is relatively easy to assess. For this reason, family history is advocated as a useful tool for identifying individuals at increased risk of disease and for tailoring preventive interventions.4,6,7 However, the challenge is to show that this knowledge has clinical utility for improving health.8Article see p 97In this issue of Circulation: Cardiovascular Genetics, Scheuner et al9 report the incremental value of adding family history information to the General Cardiovascular Risk Profile to improve identification of individuals with advanced coronary artery calcification. The incremental value of family history in disease prediction is assessed in several questions10: Does family history improve prediction beyond traditional risk factors?Does improved prediction change treatment decisions or health recommendations?Do these changes improve health outcomes or have other benefits?Do the incremental benefits outweigh the extra costs?This evaluation is a stepwise process: subsequent questions are only relevant when preceding questions have a positive answer. For example, the question of whether improved predictions change medical decision making is only relevant when the added biomarker improves prediction.Statistical methods for assessing incremental value of a risk factor for prediction and risk reclassification are evolving.11–16 One of the most frequently used measures for assessing the incremental value of a new risk factor is the c-statistic or area under the receiver operating characteristic curve (AUC),1,17 but this measure is receiving much criticism. Opponents of the AUC argue that it is not a clinically relevant way of summarizing predictive performance, that it has no intuitive clinical interpretation, and that it is naive and insensitive to change.12–14 As an alternative, measures of reclassification have been proposed, and these are now increasingly used.12,15 Yet, AUC and reclassification address different questions in the evaluation of the incremental value.The AUC assesses the discriminative accuracy of a prediction model and indicates the probability that the predicted risk of a random “patient” is higher than that of a random “nonpatient.” When the AUC of a prediction model is 1.0, predicted risks of individuals who will develop the disease are always higher than the risks of those who will not develop the disease. When the AUC is 0.5, this probability is 50%, which equals the predictive performance of tossing a coin. The AUC is particularly useful for investigating the extent to which a biomarker improves the predictive performance of a risk model. An updated model has better predictive performance than the initial model when the AUC value is higher. Also, when there is no existing risk model, the AUC can be interpreted as the improvement of the predictive performance because it is then implicitly compared with tossing a coin. The Figure shows that for every possible cut-off value, the risk model including all risk factors yielded more optimal combinations of sensitivity and specificity than the model based on genes only. When both models have similar AUCs, as also shown in the Figure, the receiver operating characteristic curves show substantial overlap implying that the models give the same combinations of sensitivity and specificity. When both models also were well calibrated, there would be no difference in the predictive performance between the 2 models. Download figureDownload PowerPointFigure. Example of receiver operating characteristic curves and AUC for the prediction of type 2 diabetes based on previously published data.18,19 The curves indicate the sensitivity and 1 specificity for every possible cut-off value. The dashed line indicates a hypothetical random predictor, which AUC=0.50. BMI indicates body mass index; FPG, fasting plasma glucose.Reclassification assesses the extent to which improvements of risk prediction models influence medical decisions.12 When prediction models are used for making treatment decisions, predicted risks often are categorized. Through updating a risk model, individuals may move to a category different from that of the initial model. The percentage of individuals that change risk categories is referred to as the percentage of reclassification. Because not every reclassification needs to be correct, alternative measures such as percentage of correct reclassification and net reclassification improvement have been developed.15,20,21 These alternative measures differ predominantly in their definition of correct reclassification. Reclassification can be considered correct not only when the observed risk in a reclassified group falls within, or is closer to, the new risk category20,21 but also when individuals who will develop the disease change to a higher risk category and those who will not, change to a lower risk category.15 Assessment of reclassification is only meaningful when clinically relevant risk thresholds are available because the percentage of reclassification depends on the values and numbers of the thresholds chosen.Scheuner et al examined 3 models for coronary artery calcium prediction, one based on the General Cardiovascular Risk Profile only (model 1) and the other 2 where family history either was added as a simple variable (model 2) or was a complex assessment (model 3). To investigate the incremental value of family history, they assessed and compared the AUC, Akaike Information Criteria, likelihood ratios, and net reclassification improvement. They found that the AUCs of the 3 models were 0.752, 0.756, and 0.759 in women and 0.718, 0.721, and 0.725 in men. These minimal differences were statistically significant according to comparisons of the likelihood ratios and Akaike Information Criteria, but these comparisons demonstrate that the updated models have a better model fit, not that they show improved prediction. In addition to the AUC, Scheuner et al examined net reclassification improvement as a measure of reclassification. They observed that model 2 reclassified 6.9% of the participants from model 1 and that model 3 reclassified 10.5% of them. The investigators concluded that incorporation of family history provides incremental prognostic value.How should we interpret the different conclusions that follow from the assessment of AUC and reclassification? The similar AUC values show that prediction is not practically improved by adding family history, but reclassification suggests that family history changes medical decisions. The answer is given by Scheuner et al. They additionally investigated the extent to which reclassification was correct—whether individuals moved in the right direction—and found that a little more than half of the individuals who were reclassified using model 2 and 3 (55% and 57.7%, respectively) were correct reclassifications. In other words, reclassification yielded approximately as many good moves as bad moves. This result is typically observed when reclassification is demonstrated in the absence of an improvement in AUC. When a risk factor does not improve prediction, reclassification simply indicates that the updated model makes different errors.Evaluating the incremental value of new risk factors is a sequential process for which AUC is a very suitable measure for a first impression about the improvement of the predictive performance of a model. When the AUC of an updated model does not differ markedly from the original model, further evaluations of reclassification, health impact, and cost-effectiveness are not warranted. Reclassification only demonstrates clinical utility if this is supported by improved predictive performance. Measures of reclassification should follow rather than replace analysis of AUC.In conclusion, it is important to assess the clinical validity and utility of family history, along with emerging biomarkers and other established risk factors for risk assessment and disease prediction. Scheuner et al are to be commended for assessing the incremental value of family history beyond other risk factors. As shown here, methods for such evaluation and their applications are still evolving. For future research, we need a strong methodologic foundation for assessing claims for improved prediction, and for answering the important question, “When does a difference make a difference?”The opinions expressed in this article are not necessarily those of the editors or of the American Heart Association.The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention.DisclosuresACJW Janssens was sponsored by the VIDI grant of the Netherlands Organisation for Scientific Research (NWO), a fellowship from the Erasmus University Medical Center and the Centre for Medical Systems Biology (CMSB) in the framework of the Netherlands Genomics Initiative (NGI).FootnotesCorrespondence to A. Cecile J.W. Janssens, PhD, Department of Epidemiology, Erasmus University Medical Center, PO Box 2040, 3000 CA Rotterdam, The Netherlands. E-mail [email protected] References 1 Tzoulaki I, Liberopoulos G, Ioannidis JP. Assessment of claims of improved prediction beyond the Framingham risk score. J Am Med Assoc. 2009; 302: 2345–2352.CrossrefMedlineGoogle Scholar2 Manolio TA, Brooks LD, Collins FS. A HapMap harvest of insights into the genetics of common disease. J Clin Invest. 2008; 118: 1590–1605.CrossrefMedlineGoogle Scholar3 Berg AO, Baird MA, Botkin JR, Driscoll DA, Fishman PA, Guarino PD, Hiatt RA, Jarvik GP, Millon-Underwood S, Morgan TM, Mulvihill JJ, Pollin TI, Schimmel SR, Stefanek ME, Vollmer WM, Williams JK. National Institutes of Health state-of-the-science conference statement: family history and improving health. Ann Intern Med. 2009; 151: 872–877.CrossrefMedlineGoogle Scholar4 Harrison TA, Hindorff LA, Kim H, Wines RC, Bowen DJ, McGrath BB, Edwards KL. Family history of diabetes as a potential public health tool. Am J Prev Med. 2003; 24: 152–159.CrossrefMedlineGoogle Scholar5 Wandeler G, Paccaud F, Vollenweider P, Waeber G, Mooser V, Bochud M. Strength of family history in predicting levels of blood pressure, plasma glucose and cholesterol. Public Health Genomics. 2009 [Epub ahead of print].Google Scholar6 Yoon PW, Scheuner MT, Peterson-Oehlke KL, Gwinn M, Faucett A, Khoury MJ. Can family history be used as a tool for public health and preventive medicine? Genet Med. 2002; 4: 304–310.CrossrefMedlineGoogle Scholar7 Guttmacher AE, Collins FS, Carmona RH. The family history–more important than ever. N Engl J Med. 2004; 351: 2333–2336.CrossrefMedlineGoogle Scholar8 Valdez R, Yoon PW, Qureshi N, Fisk JD, Green R, Khoury MJ. Family history in public health practice: a genomic tool for disease prevention and health promotion. Ann Rev Public Health. 2010.Google Scholar9 Scheuner MT, Setodji CM, Pankow JS, Blumenthal RS, Keeler E. General cardiovascular risk profile identifies advanced coronary artery calcium and is improved by family history: The Multi-Ethnic Study of Atherosclerosis. Circ Cardiovasc Genet. 2010; 3: 97–105.LinkGoogle Scholar10 Hlatky MA, Greenland P, Arnett DK, Ballantyne CM, Criqui MH, Elkind MS, Go AS, Harrell FE Jr, Hong Y, Howard BV, Howard VJ, Hsue PY, Kramer CM, McConnell JP, Normand SL, O'Donnell CJ, Smith SC Jr, Wilson PW. Criteria for evaluation of novel markers of cardiovascular risk: a scientific statement from the American Heart Association. Circulation. 2009; 119: 2408–2416.LinkGoogle Scholar11 Janes H, Pepe MS, Gu W. Assessing the value of risk predictions by using risk stratification tables. Ann Intern Med. 2008; 149: 751–760.CrossrefMedlineGoogle Scholar12 Cook NR. Use and misuse of the receiver operating characteristic curve in risk prediction. Circulation. 2007; 115: 928–935.LinkGoogle Scholar13 Pepe MS, Janes HE. Gauging the performance of SNPs, biomarkers, and clinical factors for predicting risk of breast cancer. J Natl Cancer Inst. 2008; 100: 978–979.CrossrefMedlineGoogle Scholar14 Steyerberg EW, Vickers AJ, Cook NR, Gerds T, Gonen M, Obuchowski N, Pencina MJ, Kattan MW. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology. 2010; 21: 128–138.CrossrefMedlineGoogle Scholar15 Pencina MJ, D'Agostino RBSr, D'Agostino RB Jr, Vasan RS. Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med. 2008; 27: 157–172.CrossrefMedlineGoogle Scholar16 McGeechan K, Macaskill P, Irwig L, Liew G, Wong TY. Assessing new biomarkers and predictive models for use in clinical practice: a clinician’s guide. Arch Intern Med. 2008; 168: 2304–2310.CrossrefMedlineGoogle Scholar17 Janssens AC, Van Duijn CM. Genome-based prediction of common diseases: methodological considerations for future research. Genome Med. 2009; 1: 20.CrossrefMedlineGoogle Scholar18 Lyssenko V, Almgren P, Anevski D, Orho-Melander M, Sjogren M, Saloranta C, Tuomi T, Groop L; the Botnia Study Group. Genetic prediction of future type 2 Diabetes. PLOS Med. 2005; 2: e345.CrossrefMedlineGoogle Scholar19 Janssens ACJW, Gwinn M, Subramonia-Iyer S, Khoury MJ. Does genetic testing really improve the prediction of future type 2 diabetes? PLOS Medicine. 2006; 3: e114.CrossrefMedlineGoogle Scholar20 Paynter NP, Chasman DI, Buring JE, Shiffman D, Cook NR, Ridker PM. Cardiovascular disease risk prediction with and without knowledge of genetic variation at chromosome 9p21.3. Ann Intern Med. 2009; 150: 65–72.CrossrefMedlineGoogle Scholar21 Cook NR, Ridker PM. Advances in measuring the effect of individual predictors of cardiovascular risk: the role of reclassification measures. Ann Intern Med. 2009; 150: 795–802.CrossrefMedlineGoogle Scholar Previous Back to top Next FiguresReferencesRelatedDetailsCited By Pang C, Pilkington N, Wei Y, Peters J, Roobottom C and Hyde C (2018) A methodology review on the incremental prognostic value of computed tomography biomarkers in addition to Framingham risk score in predicting cardiovascular disease: the use of association, discrimination and reclassification, BMC Cardiovascular Disorders, 10.1186/s12872-018-0777-5, 18:1, Online publication date: 1-Dec-2018. Wang X, Strizich G, Hu Y, Wang T, Kaplan R and Qi Q (2015) Genetic markers of type 2 diabetes: Progress in genome-wide association studies and clinical application for risk prediction, Journal of Diabetes, 10.1111/1753-0407.12323, 8:1, (24-35), Online publication date: 1-Jan-2016. Leening M, Steyerberg E, Van Calster B, D'Agostino R and Pencina M (2014) Net reclassification improvement and integrated discrimination improvement require calibrated models: relevance from a marker and model perspective, Statistics in Medicine, 10.1002/sim.6133, 33:19, (3415-3418), Online publication date: 30-Aug-2014. Bao W, Hu F, Rong S, Rong Y, Bowers K, Schisterman E, Liu L and Zhang C (2013) Predicting Risk of Type 2 Diabetes Mellitus with Genetic Risk Models on the Basis of Established Genome-wide Association Markers: A Systematic Review, American Journal of Epidemiology, 10.1093/aje/kwt123, 178:8, (1197-1207), Online publication date: 15-Oct-2013., Online publication date: 15-Oct-2013. Hasanaj Q, Wilson B, Little J, Montazeri Z and Carroll J (2013) Family History: Impact on Coronary Heart Disease Risk Assessment beyond Guideline-Defined Factors, Public Health Genomics, 10.1159/000353460, 16:5, (208-214), . Peeters M, van Zuilen A, van den Brand J, Bots M, Blankestijn P, Wetzels J, Blankestijn P, Bots M, van Buren M, ten Dam M, Kaasjager K, Ligtenberg G, Sijpkens Y, Henk E. S, van de Ven P, Vervoort G, Vleming L, Wetzels J and van Zuilen A (2013) Validation of the kidney failure risk equation in European CKD patients, Nephrology Dialysis Transplantation, 10.1093/ndt/gft063, 28:7, (1773-1779), Online publication date: 1-Jul-2013. Janssens A, Ioannidis J, van Duijn C, Little J and Khoury M (2011) Strengthening the Reporting of Genetic Risk Prediction Studies: The GRIPS Statement, Circulation: Cardiovascular Genetics, 4:2, (206-209), Online publication date: 1-Apr-2011. Janssens A, Ioannidis J, Bedrosian S, Boffetta P, Dolan S, Dowling N, Fortier I, Freedman A, Grimshaw J, Gulcher J, Gwinn M, Hlatky M, Janes H, Kraft P, Melillo S, O’Donnell C, Pencina M, Ransohoff D, Schully S, Seminara D, Winn D, Wright C, van Duijn C, Little J and Khoury M (2011) Strengthening the reporting of Genetic RIsk Prediction Studies (GRIPS): explanation and elaboration, Journal of Clinical Epidemiology, 10.1016/j.jclinepi.2011.02.003, 64:8, (e1-e22), Online publication date: 1-Aug-2011. Willems S, Mihaescu R, Sijbrands E, van Duijn C and Janssens A (2011) A Methodological Perspective on Genetic Risk Prediction Studies in Type 2 Diabetes: Recommendations for Future Research, Current Diabetes Reports, 10.1007/s11892-011-0235-6, 11:6, (511-518), Online publication date: 1-Dec-2011. Mihaescu R and J. W. Janssens A (2011) Two Authors Reply, American Journal of Epidemiology, 10.1093/aje/kwq485, 173:6, (714-715), Online publication date: 15-Mar-2011. Janssens A, Ioannidis J, van Duijn C, Little J and Khoury M (2011) Strengthening the reporting of genetic risk prediction studies: the GRIPS statement, European Journal of Epidemiology, 10.1007/s10654-011-9552-y, 26:4, (255-259), Online publication date: 1-Apr-2011. Khoury M, Gwinn M, Clyne M and Yu W (2011) Genetic epidemiology with a Capital E, ten years after, Genetic Epidemiology, 10.1002/gepi.20634, 35:8, (845-852), Online publication date: 1-Dec-2011. Janssens A, Ioannidis J, van Duijn C, Little J and Khoury M (2011) Strengthening the reporting of genetic risk prediction studies: the GRIPS statement, European Journal of Clinical Investigation, 10.1111/j.1365-2362.2011.02494.x, 41:9, (1004-1009), Online publication date: 1-Sep-2011. Janssens A, Ioannidis J, van Duijn C, Little J and Khoury M (2011) Strengthening the reporting of genetic risk prediction studies: the GRIPS statement, Genome Medicine, 10.1186/gm230, 3:3, (16), . Janssens A, Ioannidis J, Bedrosian S, Boffetta P, Dolan S, Dowling N, Fortier I, Freedman A, Grimshaw J, Gulcher J, Gwinn M, Hlatky M, Janes H, Kraft P, Melillo S, O’Donnell C, Pencina M, Ransohoff D, Schully S, Seminara D, Winn D, Wright C, van Duijn C, Little J and Khoury M (2011) Strengthening the reporting of genetic risk prediction studies (GRIPS): explanation and elaboration, European Journal of Clinical Investigation, 10.1111/j.1365-2362.2011.02493.x, 41:9, (1010-1035), Online publication date: 1-Sep-2011. Janssens A, Ioannidis J, van Duijn C, Little J and Khoury M (2011) Strengthening the reporting of Genetic RIsk Prediction Studies: the GRIPS Statement, Journal of Clinical Epidemiology, 10.1016/j.jclinepi.2011.02.004, 64:8, (843-847), Online publication date: 1-Aug-2011. Janssens A, Ioannidis J, Bedrosian S, Boffetta P, Dolan S, Dowling N, Fortier I, Freedman A, Grimshaw J, Gulcher J, Gwinn M, Hlatky M, Janes H, Kraft P, Melillo S, O’Donnell C, Pencina M, Ransohoff D, Schully S, Seminara D, Winn D, Wright C, van Duijn C, Little J and Khoury M (2011) Strengthening the reporting of genetic risk prediction studies (GRIPS): explanation and elaboration, European Journal of Epidemiology, 10.1007/s10654-011-9551-z, 26:4, (313-337), Online publication date: 1-Apr-2011. Janssens A, Ioannidis J, van Duijn C, Little J and Khoury M (2011) Strengthening the reporting of genetic risk prediction studies: the GRIPS statement, European Journal of Human Genetics, 10.1038/ejhg.2011.25, 19:8, (833-836), Online publication date: 1-Aug-2011. Do C, Tung J, Dorfman E, Kiefer A, Drabant E, Francke U, Mountain J, Goldman S, Tanner C, Langston J, Wojcicki A, Eriksson N and Gibson G (2011) Web-Based Genome-Wide Association Study Identifies Two Novel Loci and a Substantial Genetic Component for Parkinson's Disease, PLoS Genetics, 10.1371/journal.pgen.1002141, 7:6, (e1002141) February 2010Vol 3, Issue 1 Advertisement Article InformationMetrics https://doi.org/10.1161/CIRCGENETICS.110.938092 Originally publishedFebruary 1, 2010 Keywordsrisk assessmenteditorialpredictionreclassificationcardiovascular diseasearea under receiver operating characteristic curvefamily historyPDF download Advertisement SubjectsEpidemiology