David A. Reingold's response to Charles E. Metcalf

Abstract

Charles Metcalf provides a thoughtful overview of the challenges evaluation and policy research firms face when trying to maintain credibility and independence within the current contracting environment. We are in agreement that independence in government-supported program evaluation and policy research is a function of the selection process used to identify the firm to complete the research, as well as ownership (or control) of intellectual property produced during the course of conducting program evaluation and policy research. However, we disagree on the nature and extent to which current arrangements adequately safeguard the field of program evaluation from these threats to independence. It is inconsistent to argue that contracted program evaluation can be independent when the agency purchasing the evaluation owns (or controls) the research question and design, the method of data collection, the strategy of analysis, the data, the final report, and the rules governing the dissemination of results. In this environment, it is not clear what aspects of program evaluation conducted in this manner are independent. And the examples illustrated by Metcalf seem to reinforce the point that norms of independence in contracting evaluation are weak and must be enforced by other institutions (such as the press and various watchdog groups), because the firm conducting contract-based evaluation and policy research has little or no power relative to the client or agency purchasing the services to complete studies free from bias and interference from the agency or program being evaluated. Program evaluation is not the only domain of scientific inquiry subject to the problem of contracting bias. The long-standing struggles over the independence of medical research sponsored by the pharmaceutical and tobacco industries and environmental research sponsored by the energy industry share the same potential threats to independence as contracted evaluation and policy research. Similarly, threats to independence can be found in the auditing practices of financial firms following the Enron meltdown (among other firms that collapsed). More recently, concerns over the independence of bond rating firms and the lack of independence among property appraisers are thought to be important causes of the current financial credit crisis and the meltdown in the U.S. housing market. All of these situations share a common underlying theme: reliance on experts to produce independent facts when these experts have substantial incentives to accommodate clients' interests rather than the public good. The misalignment of incentives, combined with a lack of control (or ownership) of intellectual property, create real threats to the independence of contracted program evaluation and policy research. A defense of the status-quo arrangement is contradictory, given the importance private evaluation and policy research firms place on independence and objectivity. In other words, if these firms really care about independence and objectivity, why are they content working within the current institutional arrangements when their organizational values continually bump up against conflicting practices in the implementation of program evaluation and policy research? Perhaps short-term profits or revenue are clouding the judgment of those in the contract research firms. Maybe these firms fear that any discussion of these issues will harm long-run support for program evaluation among policymakers. Possibly there is an implicit agreement among executive branch agencies and evaluation firms that both benefit