Randomized clinical trials: Enough bang for the buck?

Abstract

Any therapy be it medical or surgical must be balanced and compared with alternative forms of treatment. For vascular surgeons the end points are often quite dramatic, frequently reinforcing the belief that the procedure is of benefit. Let us consider a few examples where the end points are not in dispute and others where serious questions can be asked. In the treatment of advanced limb ischemia, the sudden and dramatic restoration of blood flow by a surgical procedure is immediately evident both to the patient and the surgeon. A controlled clinical trial is not needed to prove that the therapy is effective in limb salvage since we have no alternative therapies that provide the same benefit. On the other hand, carotid endarterectomy—a procedure designed to remove the source of emboli and to prevent future thrombosis of the internal carotid artery (ICA)—is a slightly different story. Here the end point is not as dramatic, and one is forced to compare the incidence of subsequent ischemic events against the known natural history of the disease when conventional medical therapy is used. The uncertainty of the role of carotid endarterectomy is also clouded by two additional factors that are of importance. These include both the perioperative and postoperative mortality and morbidity rates. The benefit of the procedure may be lost if the stroke rate and mortality rate as a result of the procedure are too high. As is true with most forms of therapy, all we can use for comparison are historical controls. In this setting it is only natural that one might insist on a controlled trial to answer the question concerning the efficacy of the operation. Even the published results of the safety of the operation are often questioned because it is the surgeons themselves who tabulate their own results. These types of reports are often considered suspect. In fact, the only outcomes that are usually believed by the skeptics are the poor results that are published. In essence, the poor results are used to document the inadequacy of the operation, whereas the good results are ignored. With such uncertainties and the importance of the problem of stroke, several clinical trials are underway. These trials were designed to remove any possible bias on the part of all participants including the surgeons. In addition, they might finally define the true place of carotid endarterectomy. Unfortunately, the history of the large clinical trials has not been as definitive as one would hope. This is true for both medical and surgical trials. My experience in this field has been in part based on serving as a member of the data monitoring committee of the national study analyzing the effects of partial ileal bypass on serum cholesterol levels in the reduction of secondary prevention of cardiac events (the POSCH trial). As a member of the data monitoring committee I was privileged over a period of many years to help monitor the conduct of the trial whose principal investigator was Henry Buchwald, MD, a professor of surgery at the University of Minnesota. The chairman of the committee was Tom Chalmers, MD, a distinguished epidemiologist who has been a pioneer in the advocacy of controlled clinical trials.1Chalmers TC Celano O Sacks HS Smith H Bias treatment assignment in controlled clinical trials.N Engl J Med. 1983; 309: 1358-1361Crossref PubMed Scopus (674) Google Scholar I learned a great deal about trial design, trial conduct, and the rigorous attention to detail that is necessary to properly manage such a large effort. It appears to me that Dr. Chalmers and the committee left no stones unturned to be certain the trial was properly conducted. How then did this experience with an excellent trial affect my thinking? This will be the centerpiece of my discussion that I hope can bring a balanced view to this entire process. Barnes,2Barnes RW Understanding investigative clinical trials.J Vasc Surg. 1989; 9: 609-618Abstract Full Text Full Text PDF PubMed Scopus (21) Google Scholar in his excellent review of investigative trials, has classified the types of studies into the categories shown in Table I.Table ITypes of clinical studiesObservational Epidemiologic Cross-sectional Retrospective (case control) Prospective (cohort)Experimental Prospective (clinical trials) Controlled Randomized Nonrandomized Open table in a new tab For the purpose of this discussion I will concentrate only on the controlled trials that involve randomization because it is here that the truth will most often be expected to be found. To initiate such a trial it is necessary for the investigator to carefully follow accepted principles before such a study should commence. These are as follows:1.There has to be an identification of an unsolved clinical problem that is of sufficient importance to warrant investigation2.An adequate review of the literature should be done to ensure that the problem needs to be addressed and in what fashion3.Given the increasing use and importance of meta-analysis, it may be that the problem can be addressed by this route if there are sufficient data to permit a comparison of the available therapies. It should be noted that this is rarely feasible with surgical trials given the large numbers of patients that may be required. The most common meta-analysis that surgeons are familiar with is in the role of aspiring in the treatment of ischemic cerebrovascular events. Here it has been relatively easy to combine the results of several trials in reaching a conclusion as to efficacy4.At the outset it is necessary to ask the correct questions and formulate a hypothesis that can be tested. Here there are several examples that can be reviewed since they are pertinent to the interests of the vascular surgeon and to this discussion. Some of these are as follows:1.Lowering of the serum cholesterol levels will be effective in both the primary and secondary prevention of myocardial infarction and death;2.providing blood to the middle cerebral artery by an extracranial to intracranial bypass will reduce the risk of stroke in patients with occlusion of the ICA;3.carotid endarterectomy will, if performed with a low combined morbidity and mortality rate, be effective in reducing the incidence of stroke.4.Finally, one will have to consider the costs of such a trial and its effects on the risk-benefit scale and its cost effectiveness.3Detsky AS Are clinical trials a cost-effective investment?.JAMA. 1989; 189: 1795-1800Crossref Scopus (85) Google Scholar This consideration will also have an impact on the availability of moneys available for other clinical and basic research. It is important to realize that with few exceptions none of the clinical trials in cardiovascular disorders have ever asked, or for that matter attempted to answer, if the therapy affected the underlying disease, its rate of progression, and the changes in the lesions themselves. This is in my view a major fault of the trials—they do not provide the kind of scientific documentation that is necessary to answer fundamental questions that are of great importance. This will be dealt with later in this paper. There is no doubt that the ravages of atherosclerosis remains the major medical problem in this country. Literally thousands of studies have been performed that have related the relationship between the levels of cholesterol and the incidence of coronary artery disease. This is a question of major importance that is continually being studied. Since it is of such national importance, it is only natural that the role of lipid lowering, smoking cessation, and control of blood pressure would be addressed by large clinical trials that satisfy all of the requirements listed above. Two important examples are the Multiple Risk Factor Intervention Trial (MRFIT) and the Lipid Research Clinics Coronary Primary Prevention Trial (LRC Trial). It is not necessary to review the design or conduct of these trials since they were well designed and conducted. These are considered to be commendable examples of trial design. I would like to comment on two aspects of these giant trials. The first is the cost. The MRFIT trial cost 110 million dollars, and the LRC trial cost 150 million dollars. The sample size ultimately reached was 6400 subjects per group in the MRFIT trial and 1778 per group in the LRC trial. The estimate of risk reduction was for fatal and nonfatal coronary events: 26% for MRFIT and 36% for the LRC trial. If one were to examine the results of these two trials one would be forced to conclude the following:1.MRFIT1.The intervention did not lower the mortality rate from coronary heart disease.4Multiple risk factor intervention trial research group: risk factor changes and mortality results.JAMA. 1982; 248: 1465-1477Crossref PubMed Scopus (2323) Google Scholar2.Intervention was effective in modifying the target items, but no benefit was observed. The target items were cessation of cigarette smoking, lowering of serum cholesterol, and control of hypertension.3.An increased death rate was found in patients with hypertension with an abnormal resting electrocardiogram (ECG) who were treated with thiazide diuretics.4.LRC Trial1.Treatment with cholestyramine proved to be a very difficult medication to take with serious side effects limiting its intake.2.After 7 years the event rate in the treated group was 7% as compared to 8.6% in the control group. Mortality rate from all causes was 3.7% in the placebo group as compared to 3.6% in the cholestyramine group.5Lipid research clinics program: the lipid research clinics primary prevention trial. Results I. Reduction in incidence of coronary artery disease.JAMA. 1984; : 251-364Google Scholar, 6Lipid research clinics program: the lipid research clinics coronary primary prevention trial. Results II. The relationship of reduction in incidence of coronary heart disease to cholesterol lowering.JAMA. 1984; 251: 365-374Crossref PubMed Scopus (2074) Google Scholar3.This trial was limited to men with serum cholesterol levels above the 95% level. The difference in event rates (7% vs 8.6%) did not reach significance at the 0.01 one-sided test level that was that proposed when the trial was designed. This was changed—presumably during data analysis—to accept the 0.05 significance level (one-sided test).7Kronmal RA Commentary on the published results of the lipid research clinics: coronary primary prevention trial.JAMA. 1985; 253: 2091-2093Crossref PubMed Scopus (37) Google Scholar What does this mean and how should these studies be interpreted? If one were to take these massive and expensive clinical trials as the final test of the hypotheses, one would be forced to conclude that successful risk factor modification would have little if any effect on the death rates from coronary artery disease. The results of these trials led to extensive national debate both in the medical and lay press. In fact the National Heart, Lung, and Blood Institute (NHLBI) was forced to respond to an article by Thomas J. Moore that appeared in the Atlantic Monthly in September 1989 that seriously questioned the effort by NHLBI to mount the National Cholesterol Education Program.8Moore TJ The cholesterol myth.The Atlantic Monthly. 1989; : 37-70Google Scholar The NHLBI in their response defended the program based on the “massive body of research that has persuaded the vast majority of involved scientists around the world.”9Lenfant C The truth about cholesterol.InfoMed. December 1989; (NHLBI update)Google Scholar Clinical trials such as this are the culmination of the results of basic and applied research that led to the natural end point—a controlled examination of hypotheses. They are mounted to find the truth and influence the practice of medicine. What went wrong with these trials? Should the results be ignored, and what does it mean for future efforts in the cardiovascular area? These questions will be reviewed later. One of the more familiar trials for the vascular surgeon is the external carotid artery-internal carotid artery (ECA-ICA) bypass trial. This randomized trial was an example of how a clinical trial could, when the results were negative, essentially stop the performance of an operation for which it was intended. This trial and its results have been analyzed, reanalyzed, and criticized for its design and conduct, but the results have been widely accepted—which are that the ECA-ICA bypass does not reduce the incidence of stroke in patients with chronic occlusion of their ICA.10EC/IC bypass study group: failure of extracranial/intracranial arterial bypass to reduced the incidence of ischemic stroke.N Engl J Med. 1985; 313: 1191-1200Crossref PubMed Scopus (1557) Google Scholar My own analysis of the ECA-ICA bypass trial does not dispute the outcome, but I do believe its hypothesis was flawed. In theory, the procedure would increase blood flow to the middle cerebral artery territory and thus prevent a stroke by preventing a critical fall in brain blood flow that could lead to an ischemic event. This would fit into the hemodynamic theory of the origin of ischemic cerebrovascular events. The problem with the theory was that there was no evidence that blood flow in the middle cerebral artery territory was reduced in patients with chronic ICA occlusions. In fact, although not mentioned, the ECA artery might in some patients be the conduit for emboli to reach the middle cerebral artery! Finally, we are in the era of controlled trials examining the benefit or lack thereof, of carotid endarterectomy. These trials both here and aboard are examining the outcome of the procedure both in the asymptomatic and symptomatic patient population. A great deal of thought and review has gone into the trial design so we can be assured that the fundamental rules of the process are in place, but potential trouble is already appearing. The Veterans Administration trial of asymptomatic patients has completed it patient accrual and is now in the follow-up phase. The results of this trial will be available within the next few years. Two aspects of this trial are worrisome and could negate the results. The first is the sample size that may not have been large enough. The second is the fact that transient ischemic events are considered an end point. If the trial were positive for transient event reduction but not stroke, the results would not be accepted by some prominent neurologists. The large National Institutes of Health (NIH) studies—the trial in symptomatic patients (NASCET) and asymptomatic patients (ACAS)—are having trouble in terms of recruitment. Each of these trials is behind in patient accrual, which makes it very likely that the trial duration will have to be extended for several years beyond the targeted termination date. Although I do not have the figures on costs, these are in the “expensive” category (millions of dollars). It is also clear that much more money will have to be committed for completion. Pressures will mount to improve accrual rates—that is a natural outcome. With this added pressure, a fatigue factor always appears, and there is a tendency to loosen the entry criteria. This change in protocol could make the results suspect. Although the need for clinical trials obviously remains, I personally believe it is time to rethink the use of the large, “megabuck” trials. Are there alternatives? It is important to realize that clinical trails rarely devote any attention to the disease itself. This means that the end points used for confirmation of outcome are often fuzzy and difficult to evaluate. Although death is a most certain end point, its cause may often be difficult to ascertain with certainty. The more one begins to dissect out the causes of death in a clinical trial, the more one becomes impressed with the problems involved. The same applies to softer events where the interpretation becomes even more difficult. This is one reason that such large numbers are required to obtain satisfactory results. Although the NHLBI did not agree with the evaluation of its critics that the MRFIT and LRC trials were a failure, they certainly were forced to fall back on the “massive body of research” that preceded the trials. One important and unexpected fact to emerge in the MRFIT trial was that treatment of the patients with hypertension with thiazide diuretics led to a higher mortality rate. One must ask that if there was such a massive amount of evidence on the importance of cholesterol lowering in the origin of atherosclerosis, why were the results of the trials so disappointing? In the MRFIT trial a lower than expected mortality rate occurred in the control group. In other words, their estimated power calculations proved to be incorrect even though they were based on solid historical evidence. In addition, it is possible that the disease itself was, for reasons that are not understood, modifying itself over time. Is this an unreasonable idea? It is well known that the death rates from coronary artery disease and stroke began to fall well before there was the intense interest in cholesterol, cigarette smoking, and the treatment of hypertension. The bottom line is that the longer a trial goes on, the more likely it is that other factors not anticipated may play a role in affecting the disease in the control group. This is likely to be what happened in the control group in the MRFIT trial. Other problems appear that tend to confuse the issues and make the interpretation of the results more difficult. In the case of the LRC trial serious questions were raised about the statistical ground rules and their modification during data analysis. Since most of us are not statisticians, these arguments cannot be reconciled but leave us with a great deal of suspicion that all may not be well. Although I do not disagree with the final results of the ECA-ICA trial, it has accomplished what many clinical trials have the danger of doing. Subgroups may exist within the larger study population where the therapy that is being condemned may be effective. For example, evidence has emerged that a subset of patients exists with ICA occlusion that could benefit from an ECA-ICA bypass. These are patients whose blood flow is marginal and who have lost the ability to autoregulate in a normal fashion when flow is reduced. The potential role of ECA/ICA bypass for this subset of patients is nearly impossible to examine now given the results condemning the operation. I would suggest that this could be a negative result. The issue of the role of carotid endarterectomy is an interesting one. After all the smoke has cleared, the problem is really in my view one of safety. The review of the Rand Corporation of three geographic areas pointed out that the 30-day morbidity and mortality rate was 9.3%.11Winslow CM Solomon DH Chassin MR et al.The appropriateness of carotid endarterectomy.N Engl J Med. 1988; 318: 721-727Crossref PubMed Scopus (380) Google Scholar This is clearly too high. The operation could not be recommended if these results represented the best that could be achieved. What about safety of the operation? If one accepts the published results of carotid endarterectomy, one could assume that acceptable outcome can be achieved in both the symptomatic and symptomatic patient population. Even the skeptics would agree that a combined morbidity and mortality rate of 3% or less is likely to show benefit. This has been shown to be attainable, but the critics find this to be unacceptable since it is not the experts in this country who are doing most of the operations. The same criticism was leveled at coronary artery bypass grafting. This problem was solved when it was demonstrated that cardiac surgeons could do the operation with acceptable immediate results. This criticism of the operation is no longer heard. In fact, now we hear the opposite—the operation is too safe and thus is being more widely applied than it should be. One would hope that the surgical results in the current trials would be acceptable to all, but this remains an uncertainty. For example, if the stroke rate by the surgeons in the trials were higher than can be achieved in the community or academic centers, how would the effectiveness of the procedure be judged? Would the procedure be abandoned, or would higher surgical standards be demanded? The longer the trials go on, the fatigue and impatience factors begin to play an important role. The problems most often occur with regard to patient accrual, which is best early in the study and tends to fall off even though the trial director attempts to do otherwise. Once this occurs pressures mount from all sides. In the case of the NIH trials, the millions of dollars promised will have to be increased. This results in pressure being brought to bear on the principal investigator who may want to liberalize the entry criteria. What effect might this have? A concrete example is available and not in dispute. Every objective study that has studied the relationship between the degree of carotid stenosis and outcome has come to the same conclusion, that is, it is the high-grade carotid lesion that is the most dangerous. The corollary to this is that lesions of moderate severity—up to 50% diameter reduction—are very safe to follow with a very low annual event rate. What if an investigator attempted to increase the input to the study by including the lesser lesions? Evidence is clear that it would make the role of endarterectomy even less clear. Is this a real danger? One already hears rumors that one hopes are not true about efforts in this regard. Are there alternatives to the large, clinical trials that I am concerned about? I believe so, and I am convinced it is time to rethink this situation more completely. What is needed in my view is as follows:1.The use of objective end points that can be assessed at regular intervals over time. For example, if the goal of therapy is to prevent progression of an atherosclerotic lesion to thrombosis and total occlusion, then use a method to study it. In the context of carotid artery disease this could be done with duplex scanning. What would this mean? We cold have for the first time objective data on the effect of nonsurgical therapy on the disease itself. The numbers of patients required for such studies would be dependent on the type of patient studied and the projected rates of progression.2.Relate the outcome (progression, regression) to known risk factors that could be studied as the lesions are being followed.3.We cold finally answer some of the critical questions that remain unanswered even after millions of dollars have been spent. Are such studies feasible at reduced levels of cost and efforts? I believe so, and this is based on our own examination of the sample sizes required to answer the question as to the role of carotid endarterectomy in the treatment of the preocclusive stenoses. It is my belief that a tightly controlled study would not require the number of institutions and the complexity associated with the study of hundreds of patients. If a form of therapy could prevent the progression of an atherosclerotic lesion to total occlusion, would that have any meaning, or would it simply, as in the case of the cholesterol story, prompt the demand for the type of multimillion dollar studies I am concerned about. Finally, academic vascular surgeons will have to face the reality of clinical research in our present environment. The demand for money is at an all-time high, but its availability is less certain. The smaller, highly focused clinical trials will have to complete with the large trials for support. It is appropriate to ask if the 260 million dollars spent on the MRFIT and the LRC trials added any information that was of real benefit. The results of these trials sharpened the debate and pointed out the potential danger of such efforts. It was reliance on the large volume of evidence already available that supported the view that cholesterol lowering was beneficial. The National Cholesterol Education Program was launched in 1985. Where will we go from here? Clinical trials will and must continue. However, I believe it is time that we as surgeons ask the proper questions and enter this debate when it is reasonable to do so. Although this will require educating ourselves in areas and disciplines that are not familiar to us, we must do so.