Models: do they answer the question?

Abstract

In a model, a computer is used to simulate the process of medical decision making and associated outcomes in a given disease state by using published clinical data (disease prevalence, diagnostic test characteristics, treatment effectiveness, etc) or economic data (costs of care).The United States is currently facing a health care crisis on 2 opposing fronts. On the one hand, we must grapple with the challenge of cost containment. The United States spends approximately $2 trillion on health care each year, representing nearly 16% of the gross domestic product.1Centers for Medicare & Medicaid Services. Available at: http://www.cms.hhs.gov/NationalHealthExpendData/downloads/tables.pdf. Accessed January 30, 2008.Google Scholar These numbers become more sobering when we compare them with the health-related expenditures of other industrialized countries around the world. The United Kingdom, for instance, spends only 8% of its gross domestic product on health care while achieving similar health outcomes.2Anderson G.F. Frogner B.K. Reinhardt U.E. Health spending in OECD countries in 2004: an update.Health Aff (Millwood). 2007; 26: 1481-1489Crossref PubMed Scopus (55) Google Scholar The implication of these numbers is that our health care spending is inefficient: we are not getting our money's worth. On the other front, we are faced with the challenge of improving the quality of health care. The Institute of Medicine's 2001 report Crossing the Quality Chasm highlighted the often suboptimal quality of health care delivered in the United States.3Institute of Medicine, Committee on Quality of Health Care in AmericaCrossing the quality chasm: a new health system for the 21st century. National Academy Press, Washington, DC2001Google Scholar Yet, quality does not come without cost, and in resolving the tension between containing costs and delivering high-quality health care, we are being forced to make hard choices about what we value and where we should spend our money. The challenges are further compounded by the exponential growth of medical technology in the form of new diagnostic and therapeutic choices, which are a major influence on the rising costs of medical care.4Bodenheimer T. High and rising health care costs, 2: technologic innovation.Ann Intern Med. 2005; 142: 932-937Crossref PubMed Scopus (232) Google Scholar As a result, providers are often forced to choose between 2 therapies that have limited supporting data, each of which have differing risks and benefits. For instance, should a patient with obscure GI bleeding (OGIB) undergo initial capsule endoscopy (which is relatively safe and sensitive but provides no therapeutic options) or double-balloon enteroscopy (DBE) (which is invasive but allows for treatment)? Policy makers are faced with questions of cost versus effectiveness. Is it worth paying for up-front DBE if it is more effective but also more costly than capsule endoscopy? Such questions ask us to weigh various competing considerations, ultimately incorporating them into a single medical decision (at the provider level) or single policy recommendation (at the payer or societal level). Modeling studies provide one approach to answering these questions. In a model, a computer is used to simulate the process of medical decision making and associated outcomes in a given disease state by using published clinical data (disease prevalence, diagnostic test characteristics, treatment effectiveness, etc) or economic data (costs of care). In a model, a computer is used to simulate the process of medical decision making and associated outcomes in a given disease state by using published clinical data (disease prevalence, diagnostic test characteristics, treatment effectiveness, etc) or economic data (costs of care). A modeling study (1) systematically identifies existing clinical and economic data (“inputs”), (2) uses these data and clinical knowledge to simulate a disease process and various management strategies, (3) tabulates outcomes (typically costs and some measure of effectiveness, such as quality-adjusted life-years (QALYs), and (4) weighs (or values) these outcomes according to some prespecified criteria (cost-effectiveness, net health benefits, etc), ultimately identifying a “preferred” management strategy. Modeling studies also vary the inputs of the model within prespecified ranges and assess the impact of this variation on outcomes and conclusions (sensitivity analysis). A cost-effectiveness analysis is a particular type of model that attempts to determine whether differences in effectiveness between management strategies justify the additional cost associated with a strategy. In the current issue of this journal, Gerson and Kamal5Gerson L. Kamal A. Cost-effectiveness analysis of management strategies for obscure GI bleeding.Gastrointest Endosc. 2008; 68: 920-936Abstract Full Text Full Text PDF PubMed Scopus (99) Google Scholar have attempted to address a question of growing importance in the GI community: with the various diagnostic and treatment approaches now available for patients with OGIB, which approach is the most effective at an acceptable cost? The authors used a Markov model to determine the cost and effectiveness of several available management approaches. Incremental cost-effectiveness ratios (ICERs) were then used to summarize the results and compare the management strategies. In addition, sensitivity analyses were performed to assess the robustness of the results to the various inputs and identify inputs that particularly affected these results (hence defining potential areas for future research). The authors found that initial DBE was the most cost-effective approach in their base-case analysis, with an ICER of about $21,000 per QALY gained compared with initial push enteroscopy, an amount well within the $50,000 per QALY widely accepted as a reasonable amount to pay.6Ubel P.A. Hirth R.A. Chernew M.E. et al.What is the price of life and why doesn't it increase at the rate of inflation?.Arch Intern Med. 2003; 163: 1637-1641Crossref PubMed Scopus (513) Google Scholar They also found that initial capsule endoscopy was less effective and more costly than (“dominated” by) initial DBE. However, in sensitivity analyses, it was found that capsule endoscopy became increasingly cost-effective as the duration of follow-up was lengthened. Furthermore, the results were particularly sensitive to a number of key variables, including the quality of life (“utility”) in the well state and bleeding state and the prevalence of arteriovenous malformations (AVMs) among patients with OGIB. The authors conclude that initial DBE is a cost-effective approach for patients with OGIB, although initial capsule endoscopy may result in fewer complications and decreased endoscopic resource utilization. But does this model answer the question of which strategy is best? And can models ever really answer the question? As you might expect, the devil is in the details. Although critically appraising the results of a modeling study can be a daunting task even for the most experienced reader, every study design has its own set of validity criteria, and modeling studies are no different. In the case of modeling studies, the validity criteria are aimed at addressing 2 basic questions: (1) were the underlying assumptions of the model reasonable and (2) does the output from the model compare well to observed data? Every model must make certain fundamental assumptions regarding (1) the perspective taken (meaning who is asking the question), (2) the overall model logic and structure (including the management options that are modeled), (3) the clinical and economic inputs used, and (4) the outcomes measured. Ultimately, the validity of any model is largely a function of the validity of these various assumptions. Validity can be further confirmed by comparing model results with actual observed data (if such data are available). The perspective of a modeling study determines which outcomes are measured and, more important, how these outcomes are valued. In general, the broader the perspective, the more useful (but also more challenging to perform) the study. Ideally, a cost-effectiveness analysis should take the broad perspective of society as a whole because this will allow any “user” of the cost-effectiveness analysis to individually weigh the outcomes that they value and come to their own conclusions. In the case of the study by Gerson and Kamal, the perspective taken is that of a third-party payer. Although this perspective may be valuable to an insurer, it is less relevant to the provider and the patient. In the case of the patient, health-related outcomes (eg, QALYs) are of primary importance, and cost is of less concern. In the case of the provider, other factors may also need to be considered, including reimbursement, time engaged in managing the patient, equipment and other overhead costs, and opportunity cost (could the provider earn more money doing something else with that time, such as performing screening colonoscopies). These factors may be of particular importance for DBE, which requires substantial resources in terms of provider time and physical space. Although Gerson and Kamal did not perform formal analyses from the perspective of the patient or provider, they did provide some outcomes relevant to these groups, including QALYs and number of DBEs performed. Model logic and structure is also an important “assumption” of any modeling study. Building such a structure requires an understanding of the nuances of clinical management of a condition and the underlying disease process and its natural history. Furthermore, for a model to be complete, all reasonable management options available to the clinician need to be included (or the comparisons made between the various options may be misleading). It is also vital that all important outcomes are considered. In the study at hand, our understanding of the natural history of the disease process is quite limited, leading to several limitations in the model structure. First, the authors modeled a very short time of follow-up (1 year). Although this time horizon was extended in a sensitivity analysis, it is not known whether the available short-term data on bleeding cessation and recurrence can be reasonably extrapolated to long-term outcomes. Second, the authors limited the study to modeling a single therapeutic attempt at bleeding cessation. This design leaves out an important clinical situation that often comes up in patients with obscure GI bleeding—management of rebleeding. The impact of omitting this clinically important outcome on the model results is unclear. Alternatively, one of the strengths of the model structure is that it did include a wide array of therapeutic options, and the authors should be commended for their comprehensiveness. The results of a modeling study are also intimately connected to the specific clinical and economic data that are used in the model. In the case of this study, the available data on diagnostic test accuracy, the effectiveness of the various treatment options, and the costs of care (especially for DBE) are quite limited. Many of these data are based on small individual studies with significant heterogeneity in study design and results. The authors attempt to address these limitations through sensitivity analyses. The 1-way sensitivity analyses (varying individual variables, one at a time, while all other variables are held constant at their base-case value) are this study's strong suit. Although the base-case analysis may be of limited value because of the uncertainty in the input values, the sensitivity analyses allow us to define some areas that are worthy of additional study. The results suggest that future research should better define the quality of life in patients with recurrent or chronic GI bleeding and that we should also develop means to better predict which patients are likely to have small intestinal AVMs. On the other hand, Monte Carlo (or multivariate) sensitivity analysis, which is widely used in the medical literature as well as in this particular study, sounds attractive in theory but is fraught with methodologic quandaries in practice. In a Monte Carlo analysis, the cost-effectiveness analysis is repeated multiple times (typically at least 1000 times) by using values selected at random for each variable (as if being chosen by a roulette wheel in Monte Carlo) to determine the percentage of “runs” in which each strategy is preferred. But unlike a roulette wheel, where each number (value) always has an equal chance of being selected (a uniform distribution), variables in the natural world tend to be normally distributed (a bell curve) or skewed toward one end of the range, so that some values may be more likely to be selected than others. A Monte Carlo analysis is attractive because it allows us to get a sense of the overall impact of uncertainty on the output of the model. However, such an analysis presupposes not only that we know the range of plausible values for each variable but also that we know the exact shape of the underlying distribution of each variable—an assumption that is almost never true in medical research. Furthermore, it assumes we know precisely how changes in the value of one variable affect the value of another variable. For instance, if the sensitivity of capsule endoscopy for small-bowel AVMs is randomly set at 80% rather than 87% as the authors initially assumed, one might reasonably expect that the sensitivity of DBE may also be lower than initially assumed. However, we have no way of knowing precisely how closely these changes ought to parallel each other. As a result of these limitations in available data, Monte Carlo analysis can become an imprecise method cloaked in the guise of statistically sound principles. Nevertheless, Monte Carlo analysis has become an expected analysis in medical cost-effectiveness studies. Despite this expectation, we feel that Monte Carlo analyses provide false assurance of the precision of results. We are much more interested in the results of 1-way sensitivity analyses. Although perspective, model structure, and input variables largely define the output of a model, models go one step further and weigh these outputs in a prespecified way. In the case of this study, the outcomes measured include costs and QALYs. However, as the authors point out, the quality of life in patients with GI bleeding has not been well quantified. Furthermore, it is well described that patients with a chronic health condition that has an acute onset (such as spinal cord injury) adapt to their illnesses over time.7Westgren N. Levi R. Quality of life and traumatic spinal cord injury.Arch Phys Med Rehabil. 1998; 79: 1433-1439Abstract Full Text PDF PubMed Scopus (374) Google Scholar Using the QALY measure for acute GI bleeding may therefore underestimate the value of life with continuing OGIB (biasing the results toward an intensive diagnostic and treatment strategy). Despite these shortcomings, however, we must still find a way to make the difficult choice of whether to choose initial DBE over capsule endoscopy, and the study of Drs Gerson and Kamal provides a quantitative answer that incorporates the available clinical and economic data on this topic. In such instances, we must keep in mind that there may be gaps between the “perfect” study, the best study that can be performed, and the best study that has been performed. In the case of management of OGIB, the Gerson study may represent the best data that we currently have. However, we can do better in the future. It would not be inconceivable to perform a randomized controlled trial comparing the various treatment strategies studied in the model, and such an approach would provide better evidence of the costs, risks, and benefits of each approach. Until that time, however, clinicians will need to make decisions about which approach to use in patients with OGIB, and the effort by Gerson and Kamal in synthesizing the various data on this topic may aid physicians faced with such management decisions. In summary, models have an important role in addressing questions of choice in our increasingly complex health care system, and they are well suited to address questions that are difficult to answer through other traditional research designs. As with any study design, such studies should be critically appraised with standardized validity criteria. It should also be kept in mind that the ideal study for a particular question cannot always be performed and the data from the best study that can be performed may not yet be available. However, we must still make decisions regarding how to care for our patients, and sometimes the best available evidence will come from a modeling study. Understanding and appreciating the role of such studies in the medical literature is not just an intellectual exercise. Models can indeed answer the question, both benefitting our patients and informing our health policies. The authors report that there are no disclosures relevant to this publication. J. H. Rubenstein is the Damon Runyon Cancer Research Foundation Gordon Family Clinical Investigator and is supported by National Institutes of Health grant No. NIDDK 1K23DK079291-01.