What is the Future of Mortality Prediction Models in Heart Valve Surgery?

Abstract

Doctor Jin and associates focus on validation and improvement of logistic regression models for the prediction of early mortality after heart valve surgery in this issue of The Annals of Thoracic Surgery [1Jin R. Grunkemeier G.L. Starr A. Providence Health System Cardiovascular Study GroupValidation and refinement of mortality risk models for heart valve surgery.Ann Thorac Surg. 2005; 80: 471-479Abstract Full Text Full Text PDF PubMed Scopus (48) Google Scholar]. The intent of this brief editorial is to place their work and risk prediction models in general into the context of the day-to-day demands of cardiac surgery practice for reliable patient counseling and surgeon accountability for adverse outcomes.The authors reviewed previously published models from several large series. They examined one of them, the Northern New England Cardiovascular Disease Study Group (NNECDSG) prediction models [2Nowicki E.R. Birkmeyer N.J. Weintraub R.W. et al.Northern New England Cardiovascular Disease Study G, the Center for Evaluative Clinical Sciences DMS. Multivariable prediction of in-hospital mortality associated with aortic and mitral valve surgery in Northern New England.Ann Thorac Surg. 2004; 77: 1966-1977Abstract Full Text Full Text PDF PubMed Scopus (164) Google Scholar] for early mortality after heart valve surgery, to validate their transportability and effectiveness of performance. The original purpose of the NNECDSG models was to identify critical risk factors associated with early mortality and to bring these to the attention of collaborating clinicians for risk assessment and patient informed consent. Both a scorecard method and the equation of the model for estimating early mortality were presented. The combined aortic and mitral NNECDSG series numbered 8,943 cases (5,793 aortic and 3,150 mitral) during the period from 1991 through 2001, which included a time variable to account for variation in experience with time. The Provident Health System (PHS) combined series numbered 4,920 cases (3,324 aortic and 1,596 mitral) during the more recent period from 1997 through March 2004.The PHS and NNECDSG series were similar in the distribution of risk factors and the effects they had on short-term mortality. The NNECDSG models were adjusted to the most current era (1999 to 2001), and performed well in predicting mortality for both valve sites in the PHS patients. This suggests that the NNECDSG models as published would have been reliable clinical tools for patient counseling and risk adjustment in the PHS series.The authors then developed a new model by studying their own dataset, using some different modeling strategies for handling continuous variables and missing data. The result was a single PHS model with improved discrimination and similar, acceptable calibration for both valve sites. In forming the single model, combining both aortic and mitral operative positions, the authors explain that operations on these two valve positions share similar risk factors, as their review of the literature supports.It is important to note, however, that the stated difference for aortic and mitral valve mortality in the PHS series was only 1.7% (6.0% mitral and 4.3% aortic). In the NNECDSG series this difference was 3.2% (9.4% mitral and 6.2% aortic). The power to predict this difference in a model depends on the magnitude of the difference as well as the number of events and cases in the series used for model development. For the larger NNECDSG series with a greater difference in mortality by valve position, a two-model approach was statistically feasible with good model performance for aortic and mitral valve positions. In addition, the NNECDSG mitral model retained a variable for replacement versus repair of the mitral valve, allowing procedure-specific risk prediction.The most recent version of The Society of Thoracic Surgeons’ (STS) risk models [3Edwards F.H. Peterson E.D. Coombs L.P. et al.Prediction of operative mortality after valve replacement surgery.J Am Coll Cardiol. 2001; 37: 885-892Abstract Full Text Full Text PDF PubMed Scopus (353) Google Scholar] using a large series of patients combined cases for aortic and mitral valve positions and created two models, one for isolated valve operation and another for valve operation with coronary bypass graft surgery (CABG). A variable for valve position and interaction terms were retained in The STS models. Using the New York State Cardiac Surgery Reporting System (CSRS), Hannan and associates [4Hannan E.L. Racz M.J. Jones R.H. et al.Predictors of mortality for patients undergoing cardiac valve replacements in New York State.Ann Thorac Surg. 2000; 70: 1212-1218Abstract Full Text Full Text PDF PubMed Scopus (84) Google Scholar] published separate models by aortic and mitral position and corresponding models for concomitant CABG. Both The STS and CSRS series reported substantial mortality differences across categories of valve position and concomitant CABG. Thus, three of the largest studies published presented models permitting risk prediction and risk adjustment for the two valve positions.The PHS model did not have the power to support a variable for valve position. Nevertheless, the result is a risk prediction model calibrated to regional experience that can be used for patient informed consent and evaluation of variations in outcomes by collaborating surgeons and medical centers. It would not be sensitive, however, to the small mortality difference between aortic or mitral valve surgery patients in their experience.A modeling approach permitting discrimination of risk between the two valve positions acknowledges the intuitive notion that patients with aortic valve disease may differ from those with mitral valve disease in important pathophysiologic respects. These differences may be associated with variation in outcomes. The obvious differences in early mortality between valve positions in The STS, CSRS, and NNECDSG cohorts support this contention.The previously mentioned studies vary in size from large cohorts at the state and national levels to smaller regional ones, but together they form a strong consensus about the profile of risk factors to be considered in any risk assessment model for early mortality after heart valve surgery. The core group of factors is similar. These include patient characteristics and comorbid disease, acuity of clinical condition and intended surgical procedure (valve position with or without concomitant CABG). Data for these factors are known before surgery and can be standardized reasonably well across institutions. Although the strength of the effect of risk factors may vary among study series, the direction of effect is the same, indicating a constancy of biologic action. Not surprisingly, many of these same risk factors are generic to any cardiac operation including coronary bypass grafting surgery [5Jones R.H. Hannan E.L. Hammermeister K.E. et al.Identification of preoperative variables needed for risk adjustment of short-term mortality after coronary artery bypass graft surgery. The working group panel on the Cooperative CABG Database Project.J Am Coll Cardiol. 1996; 28: 1478-1487Abstract Full Text PDF PubMed Scopus (247) Google Scholar]. Performance of the various heart valve surgery models is comparable and reasonably good, as evaluated by statistics for discrimination and goodness of fit.Although most of the models discussed have much in common, it seems unlikely that one “best model” for all purposes and all clinical groups will be found. Given the variation in patient care processes by surgeons and across institutions, one would expect that risk factor presence and strength of effect would also vary in models developed from different cohorts. Therefore a national-level model, such as that of the STS, may not be as effective for certain regional or local uses as a model developed from a narrower experience. For example, models (such as those of the PHS or NNECDSG) reflecting regional or local experience would likely be more accurate for patient-specific risk determination and reliable informed consent. On the other hand, the STS model permits institutions and regional collaborations to make risk-adjusted comparisons with a benchmark, the national average mortality.Measuring and reporting adjusted mortality is, however, only the first step toward ultimate improvement in results. Even the best performing of the previously described models explains only a limited amount of the variability of outcomes. There is a substantially higher early mortality for heart valve surgery compared with isolated CABG, as well as considerable mortality variation across categories of valve operative site and concomitant CABG. This knowledge should stimulate the examination of what risk factors should be studied if both early and long-term results for heart valve surgery are to be improved in the dramatic way that has occurred for CABG during the last decade. For example, some preoperative variables in current models may be surrogates for as yet undiscovered or unaddressed factors that have implications in redefining patient care processes. This is also true for factors that represent intraoperative and postoperative processes of care.To date, most of the emphasis in the use of prediction models has been for risk adjustment and benchmarking in quality assessment efforts. Patients have become increasingly sophisticated in their knowledge of medical matters. They demand information tailored to their specific condition and what treatment they face. Together with clinicians, patients are not just interested in early results, but also in prognosis beyond the hospital stay as a measure of success of the heart valve surgery.How can the next generation of improvement in model performance and utility for patient-specific risk prediction be achieved? One approach is to consider additional valve disease-specific risk factors thus far neglected in multivariable model development, and extend the outcome to include long-term all-cause mortality. In the treatment of valvular heart disease, the appropriateness and timing of operation relates not only to the level of symptoms, but also to the degree of ventricular morphologic and functional change.Left ventricular dimensions, wall thickness, and their derivations (ventricular volume and mass) are some specific examples of measures that quantify preoperative changes in ventricle morphology due to the varying hemodynamic burden of the various forms of valvular heart disease. Measures for these factors were notably absent from all current valve risk models. Many small detailed studies have shown the association of these factors with survival after heart valve surgery, and formed the basis for the American Heart Association/American College of Cardiology (AHA/ACC) Guidelines for the Treatment of Patients with Valvular Heart Disease [6Bonow R.O. Carabello B. de Leon Jr, A.C. et al.Guidelines for the management of patients with valvular heart disease: executive summary. A report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines (Committee on Management of Patients with Valvular Heart Disease).Circulation. 1998; 98: 1949-1984Crossref PubMed Scopus (812) Google Scholar]. Clinicians consider these readily available echocardiographic measures when counseling patients prior to surgery, and models should ideally reflect and support this approach in the decision-making process. All-cause mortality data are available by a probabilistic match with the National Death Index [7Williams BC, Demitrack LB, Fries BE. The accuracy of the National Death Index when personal identifiers other than social security number are used. Am J Public Health 1145;82:1145–7.Google Scholar]. Parametric survival methods that handle risk factor information are well developed and can generate equations for patient-specific prediction of both early and late survival [8Blackstone E. Generating knowledge from information, data, and analyses.in: Kouchoukos N.T. Blackstone E.H. Doty D.B. Hanley F.L. Karp R.B. Kirklin/Barratt-Boyes Cardiac Surgery. 3rd ed. Churchill Livingstone, Philadelphia2003: 254-350Google Scholar].On the scale of the large cohorts necessary for multivariable model development, inclusion of these additional data potentially poses costly data management challenges. Nevertheless, the resultant tendency to collect a limited number of risk factors for more parsimonious models should be resisted. Models that specify less are less useful. As computer-based medical records become more common and the demands for better outcomes encompass long-term results, these additional sets of variables for the purposes previously outlined may be considered part of the new core risk factor profile for heart valve surgery risk models in the future.The recognition that variation in outcomes after cardiac surgery is an opportunity for improvement of patient care is a firmly established principle of quality assessment. Thoroughly applied, this process currently considers the broad scope of risk-adjusted early mortality, complication rates, adherence to evidence-based processes of care, appropriateness of operation as suggested by the AHA/ACC Guidelines for the treatment of patients with valvular or coronary heart disease, and documentation of patient informed consent. These various aspects of quality assessment are related to each other in that the poor outcomes of excess mortality or morbidity often have their basis in combinations of inadequate processes of care, inappropriate surgery, and poor patient-doctor decision-making.In conclusion, the authors have made an important contribution to the dialogue in the literature regarding mortality prediction models for valvular heart surgery patients by the transparency of their methods and their reporting of model specifications, including all coefficients. Mortality risk prediction models for heart valve surgery deal with only one aspect of the total quality assessment process. Alone they do not determine the acceptability of results. Their uses are not only limited to risk adjustment comparisons and benchmarking, they are also an indicator of risk factors associated with an adverse outcome and help generate hypotheses to improve patient care. They can enhance objective, patient-specific estimation of early risk and long-term prognosis. Increasing the future performance and utility of these models will depend on studying a greater depth of process and pathophysiologic information. In the end, however, these models are tools to aid quality assessment, process improvement, patient counseling and decision-making, not substitutes for clinical judgment. Doctor Jin and associates focus on validation and improvement of logistic regression models for the prediction of early mortality after heart valve surgery in this issue of The Annals of Thoracic Surgery [1Jin R. Grunkemeier G.L. Starr A. Providence Health System Cardiovascular Study GroupValidation and refinement of mortality risk models for heart valve surgery.Ann Thorac Surg. 2005; 80: 471-479Abstract Full Text Full Text PDF PubMed Scopus (48) Google Scholar]. The intent of this brief editorial is to place their work and risk prediction models in general into the context of the day-to-day demands of cardiac surgery practice for reliable patient counseling and surgeon accountability for adverse outcomes. The authors reviewed previously published models from several large series. They examined one of them, the Northern New England Cardiovascular Disease Study Group (NNECDSG) prediction models [2Nowicki E.R. Birkmeyer N.J. Weintraub R.W. et al.Northern New England Cardiovascular Disease Study G, the Center for Evaluative Clinical Sciences DMS. Multivariable prediction of in-hospital mortality associated with aortic and mitral valve surgery in Northern New England.Ann Thorac Surg. 2004; 77: 1966-1977Abstract Full Text Full Text PDF PubMed Scopus (164) Google Scholar] for early mortality after heart valve surgery, to validate their transportability and effectiveness of performance. The original purpose of the NNECDSG models was to identify critical risk factors associated with early mortality and to bring these to the attention of collaborating clinicians for risk assessment and patient informed consent. Both a scorecard method and the equation of the model for estimating early mortality were presented. The combined aortic and mitral NNECDSG series numbered 8,943 cases (5,793 aortic and 3,150 mitral) during the period from 1991 through 2001, which included a time variable to account for variation in experience with time. The Provident Health System (PHS) combined series numbered 4,920 cases (3,324 aortic and 1,596 mitral) during the more recent period from 1997 through March 2004. The PHS and NNECDSG series were similar in the distribution of risk factors and the effects they had on short-term mortality. The NNECDSG models were adjusted to the most current era (1999 to 2001), and performed well in predicting mortality for both valve sites in the PHS patients. This suggests that the NNECDSG models as published would have been reliable clinical tools for patient counseling and risk adjustment in the PHS series. The authors then developed a new model by studying their own dataset, using some different modeling strategies for handling continuous variables and missing data. The result was a single PHS model with improved discrimination and similar, acceptable calibration for both valve sites. In forming the single model, combining both aortic and mitral operative positions, the authors explain that operations on these two valve positions share similar risk factors, as their review of the literature supports. It is important to note, however, that the stated difference for aortic and mitral valve mortality in the PHS series was only 1.7% (6.0% mitral and 4.3% aortic). In the NNECDSG series this difference was 3.2% (9.4% mitral and 6.2% aortic). The power to predict this difference in a model depends on the magnitude of the difference as well as the number of events and cases in the series used for model development. For the larger NNECDSG series with a greater difference in mortality by valve position, a two-model approach was statistically feasible with good model performance for aortic and mitral valve positions. In addition, the NNECDSG mitral model retained a variable for replacement versus repair of the mitral valve, allowing procedure-specific risk prediction. The most recent version of The Society of Thoracic Surgeons’ (STS) risk models [3Edwards F.H. Peterson E.D. Coombs L.P. et al.Prediction of operative mortality after valve replacement surgery.J Am Coll Cardiol. 2001; 37: 885-892Abstract Full Text Full Text PDF PubMed Scopus (353) Google Scholar] using a large series of patients combined cases for aortic and mitral valve positions and created two models, one for isolated valve operation and another for valve operation with coronary bypass graft surgery (CABG). A variable for valve position and interaction terms were retained in The STS models. Using the New York State Cardiac Surgery Reporting System (CSRS), Hannan and associates [4Hannan E.L. Racz M.J. Jones R.H. et al.Predictors of mortality for patients undergoing cardiac valve replacements in New York State.Ann Thorac Surg. 2000; 70: 1212-1218Abstract Full Text Full Text PDF PubMed Scopus (84) Google Scholar] published separate models by aortic and mitral position and corresponding models for concomitant CABG. Both The STS and CSRS series reported substantial mortality differences across categories of valve position and concomitant CABG. Thus, three of the largest studies published presented models permitting risk prediction and risk adjustment for the two valve positions. The PHS model did not have the power to support a variable for valve position. Nevertheless, the result is a risk prediction model calibrated to regional experience that can be used for patient informed consent and evaluation of variations in outcomes by collaborating surgeons and medical centers. It would not be sensitive, however, to the small mortality difference between aortic or mitral valve surgery patients in their experience. A modeling approach permitting discrimination of risk between the two valve positions acknowledges the intuitive notion that patients with aortic valve disease may differ from those with mitral valve disease in important pathophysiologic respects. These differences may be associated with variation in outcomes. The obvious differences in early mortality between valve positions in The STS, CSRS, and NNECDSG cohorts support this contention. The previously mentioned studies vary in size from large cohorts at the state and national levels to smaller regional ones, but together they form a strong consensus about the profile of risk factors to be considered in any risk assessment model for early mortality after heart valve surgery. The core group of factors is similar. These include patient characteristics and comorbid disease, acuity of clinical condition and intended surgical procedure (valve position with or without concomitant CABG). Data for these factors are known before surgery and can be standardized reasonably well across institutions. Although the strength of the effect of risk factors may vary among study series, the direction of effect is the same, indicating a constancy of biologic action. Not surprisingly, many of these same risk factors are generic to any cardiac operation including coronary bypass grafting surgery [5Jones R.H. Hannan E.L. Hammermeister K.E. et al.Identification of preoperative variables needed for risk adjustment of short-term mortality after coronary artery bypass graft surgery. The working group panel on the Cooperative CABG Database Project.J Am Coll Cardiol. 1996; 28: 1478-1487Abstract Full Text PDF PubMed Scopus (247) Google Scholar]. Performance of the various heart valve surgery models is comparable and reasonably good, as evaluated by statistics for discrimination and goodness of fit. Although most of the models discussed have much in common, it seems unlikely that one “best model” for all purposes and all clinical groups will be found. Given the variation in patient care processes by surgeons and across institutions, one would expect that risk factor presence and strength of effect would also vary in models developed from different cohorts. Therefore a national-level model, such as that of the STS, may not be as effective for certain regional or local uses as a model developed from a narrower experience. For example, models (such as those of the PHS or NNECDSG) reflecting regional or local experience would likely be more accurate for patient-specific risk determination and reliable informed consent. On the other hand, the STS model permits institutions and regional collaborations to make risk-adjusted comparisons with a benchmark, the national average mortality. Measuring and reporting adjusted mortality is, however, only the first step toward ultimate improvement in results. Even the best performing of the previously described models explains only a limited amount of the variability of outcomes. There is a substantially higher early mortality for heart valve surgery compared with isolated CABG, as well as considerable mortality variation across categories of valve operative site and concomitant CABG. This knowledge should stimulate the examination of what risk factors should be studied if both early and long-term results for heart valve surgery are to be improved in the dramatic way that has occurred for CABG during the last decade. For example, some preoperative variables in current models may be surrogates for as yet undiscovered or unaddressed factors that have implications in redefining patient care processes. This is also true for factors that represent intraoperative and postoperative processes of care. To date, most of the emphasis in the use of prediction models has been for risk adjustment and benchmarking in quality assessment efforts. Patients have become increasingly sophisticated in their knowledge of medical matters. They demand information tailored to their specific condition and what treatment they face. Together with clinicians, patients are not just interested in early results, but also in prognosis beyond the hospital stay as a measure of success of the heart valve surgery. How can the next generation of improvement in model performance and utility for patient-specific risk prediction be achieved? One approach is to consider additional valve disease-specific risk factors thus far neglected in multivariable model development, and extend the outcome to include long-term all-cause mortality. In the treatment of valvular heart disease, the appropriateness and timing of operation relates not only to the level of symptoms, but also to the degree of ventricular morphologic and functional change. Left ventricular dimensions, wall thickness, and their derivations (ventricular volume and mass) are some specific examples of measures that quantify preoperative changes in ventricle morphology due to the varying hemodynamic burden of the various forms of valvular heart disease. Measures for these factors were notably absent from all current valve risk models. Many small detailed studies have shown the association of these factors with survival after heart valve surgery, and formed the basis for the American Heart Association/American College of Cardiology (AHA/ACC) Guidelines for the Treatment of Patients with Valvular Heart Disease [6Bonow R.O. Carabello B. de Leon Jr, A.C. et al.Guidelines for the management of patients with valvular heart disease: executive summary. A report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines (Committee on Management of Patients with Valvular Heart Disease).Circulation. 1998; 98: 1949-1984Crossref PubMed Scopus (812) Google Scholar]. Clinicians consider these readily available echocardiographic measures when counseling patients prior to surgery, and models should ideally reflect and support this approach in the decision-making process. All-cause mortality data are available by a probabilistic match with the National Death Index [7Williams BC, Demitrack LB, Fries BE. The accuracy of the National Death Index when personal identifiers other than social security number are used. Am J Public Health 1145;82:1145–7.Google Scholar]. Parametric survival methods that handle risk factor information are well developed and can generate equations for patient-specific prediction of both early and late survival [8Blackstone E. Generating knowledge from information, data, and analyses.in: Kouchoukos N.T. Blackstone E.H. Doty D.B. Hanley F.L. Karp R.B. Kirklin/Barratt-Boyes Cardiac Surgery. 3rd ed. Churchill Livingstone, Philadelphia2003: 254-350Google Scholar]. On the scale of the large cohorts necessary for multivariable model development, inclusion of these additional data potentially poses costly data management challenges. Nevertheless, the resultant tendency to collect a limited number of risk factors for more parsimonious models should be resisted. Models that specify less are less useful. As computer-based medical records become more common and the demands for better outcomes encompass long-term results, these additional sets of variables for the purposes previously outlined may be considered part of the new core risk factor profile for heart valve surgery risk models in the future. The recognition that variation in outcomes after cardiac surgery is an opportunity for improvement of patient care is a firmly established principle of quality assessment. Thoroughly applied, this process currently considers the broad scope of risk-adjusted early mortality, complication rates, adherence to evidence-based processes of care, appropriateness of operation as suggested by the AHA/ACC Guidelines for the treatment of patients with valvular or coronary heart disease, and documentation of patient informed consent. These various aspects of quality assessment are related to each other in that the poor outcomes of excess mortality or morbidity often have their basis in combinations of inadequate processes of care, inappropriate surgery, and poor patient-doctor decision-making. In conclusion, the authors have made an important contribution to the dialogue in the literature regarding mortality prediction models for valvular heart surgery patients by the transparency of their methods and their reporting of model specifications, including all coefficients. Mortality risk prediction models for heart valve surgery deal with only one aspect of the total quality assessment process. Alone they do not determine the acceptability of results. Their uses are not only limited to risk adjustment comparisons and benchmarking, they are also an indicator of risk factors associated with an adverse outcome and help generate hypotheses to improve patient care. They can enhance objective, patient-specific estimation of early risk and long-term prognosis. Increasing the future performance and utility of these models will depend on studying a greater depth of process and pathophysiologic information. In the end, however, these models are tools to aid quality assessment, process improvement, patient counseling and decision-making, not substitutes for clinical judgment. Validation and Refinement of Mortality Risk Models for Heart Valve SurgeryThe Annals of Thoracic SurgeryVol. 80Issue 2PreviewThe Northern New England Cardiovascular Disease Study Group (NNE) recently published risk models for hospital mortality after heart valve surgery. The Providence Health System Cardiovascular Study Group (PHS) has been collecting similar heart valve data for 8 years, providing an ideal opportunity to both validate the NNE risk models and attempt to produce an improved model, by using some different modeling techniques. Full-Text PDF