Article-at-a-Glance Background Many health care organizations–including most managed care organizations (MCOs)–measure the success of quality improvement (QI) interventions using the before–after comparison. However, before–after comparisons may lead to false conclusions about the efficacy of a QI intervention, reflecting measurement bias, confounding due to extraneous variables, regression toward the mean, and secular trends. Performing a randomized controlled trial (RCT) of a QI intervention may result in net cost savings to the organization, as demonstrated in an example of a QI intervention to increase the use of inhaled steroids by persons with moderate to severe asthma. Methods Decision analysis and probability theory were applied to calculate the expected financial impact of three approaches to the design and evaluation of QI interventions. One approach (the strategy) involved the standard before-after measurement. The other two approaches involved RCTs of slightly different configurations, one in which the majority receives the intervention and the other in which only a minority receives the intervention. Results The expected net costs to the health care organization of each of the three QI intervention and evaluation strategies were estimated, assuming that the intervention increases inhaled steroid use. The optimal strategy varied, depending on the predicted effectiveness of the intervention. For example, if the intervention had no true effect on the rate of inhaled steroid use among the target patients with asthma, then the intervene all strategy was most expensive, at $191,000, for the two-year cycle. The some strategy, in which a small group (250 people) receives the intervention and the remainder do not, was the least expensive, at $91,000. If there is a relatively high degree of intervention effectiveness, say a 10% increase in steroid use, the intervene all strategy was preferable, with net costs of only $37,500. Discussion If the suspected effectiveness of a QI intervention is low (or unproven), MCOs can save money by testing the intervention using a small RCT first before applying it to all members of their targeted populations. On the other hand, if the intervention is suspected to be very effective (on the basis of either support from the literature or prior experience with similar interventions), there are no savings from performing a randomized trial. Conclusions The QI process itself can be improved by more frequent use of RCTs and by the use of decision analytic tools to decide when an RCT is indicated. Initial QI efforts may be highly effective, but subsequent efforts in the same population may produce smaller increments. The RCT methodology is crucial to determine if small changes observed are actually due to the intervention and not other factors, such as secular trends. Routine use of RCTs to determine the effectiveness of alternative QI interventions forms a statistical process control framework through which to continually reevaluate the nature and scope of QI efforts.