How Research Influences Policy Makers: Still Hazy After All These Years

Abstract

Libraries have been written about the theory and practice of public policy making. Yet, this enormous scholarship has proved insufficient to lift the veil of mystery and idiosyncrasy that shrouds the art of decision making. The heady ambition to turn both clinical practice and health policy into evidence-based bastions of rationalist decision making has been downgraded; the vocabulary is now “evidence-informed,” and the realm of admissible evidence has been greatly expanded to include preferences, political contingencies, and psychology (1). This newfound conceptual modesty and nuance does not suggest that we should abandon efforts to understand decision-making processes and to enhance the role of research-based evidence in policy. It merely confirms the complexity, contingency, and messiness of the terrain. In this issue of the Journal, Brownson et al. (2) have boldly entered this murky territory. They attempt to identify the potential impact of carefully crafted policy briefs on three groups of policy makers: legislators, legislative committee staffers, and public health executives. They reviewed a wide literature and created four policy briefs—data- or story-focused briefs that included state- or local-level data—on the topic of mammography screening. They persisted with recruitment efforts to engage hundreds of participants in six states, and given the topic and the terms of engagement, the 35% response rate is impressive. Their analytic objective was to identify which approaches are most effective in which circumstances and which characteristics of both the policy briefs and their recipients predict influence and uptake. It is therefore a joyless duty to report that for all its creativity and effort, the study is less of an advance in our understanding of how evidence, mode of presentation, and policy making interact than one would have hoped. Its ambitions were not fully realized because of some critical design limitations that raise serious doubts about what the findings tell us, and whether they rest on a solid foundation. The first serious design issue is the choice of subject matter. If one wants to perform an independent test of policy brief types among related but different audiences, it would be wise to choose a topic that respondents are unlikely to be informed about or invested in. It is unimaginable that a substantial proportion of respondents were unaware of mammography screening and/or neutral as to its utility. If the briefs had been designed to present the different perspectives on mammography, the study could have revealed how preexisting views and conclusions affect the reception of potentially countervailing information. Instead, all of the briefs promote the same entirely pro-mammography perspective, but nothing about the hazards of interventions occasioned by ambiguous screening results or any acknowledgment that while there may be policy consensus that screening women older than 50 years is a sound strategy, scientific controversy persists. One can counter that it is of no consequence whether the substantive content of the policy briefs is contestable—the study was designed to elicit views on clarity and use, not on scientific accuracy or comprehensive ness—but the very prominence of mammography in public discourse is an avoidable contaminant in a laboratory already fraught with uncertainty. This leads to a second issue: the definition of “use.” The ques tionnaires simply asked whether the respondents intended to use the brief. We do not know what “use” means. Does it mean that the brief created or altered either their understanding of the issue or their inclination to affect or create policy? Clear and persuasive information can confirm or change opinion about what to do and the urgency of doing it. Understanding how the policy brief “works” requires some understanding of respondents’ baseline views and how the new or repackaged information affects them. It also requires knowledge of other sources of information and advocacy on the mammography issue. For some, the policy brief may be the first and only thing they have ever read on the subject; for others, it may be the first effective compilation of evidence in a single accessible format; and for others, it may be entirely redundant. Yet, the already well informed may report intention to use the briefs because they are already motivated to pursue policy in the area and are happy to have another platoon in the supporting army.