Evidence-based Policy Movement Research Articles

A model of faulty and faultless disagreement for post-hoc assessments of knowledge utilization in evidence-based policymaking

When evidence-based policymaking is so often mired in disagreement and controversy, how can we know if the process is meeting its stated goals? We develop a novel mathematical model to study disagreements about adequate knowledge utilization, like those regarding wild horse culling, shark drumlines and facemask policies during pandemics. We find that, when stakeholders disagree, it is frequently impossible to tell whether any party is at fault. We demonstrate the need for a distinctive kind of transparency in evidence-based policymaking, which we call transparency of reasoning. Such transparency is critical to the success of the evidence-based policy movement, as without it, we will be unable to tell whether in any instance a policy was in fact based on evidence.

Analysing the information sources Brazilian bureaucrats use as evidence in everyday policymaking

The evidence-based policy (EBP) movement argues for policy actors, especially public officials, to use scientific evidence on what works to improve public policies. Empirical research in Anglo-Saxon countries shows that public officials do not use scientific knowledge that widely, often preferring other sources of information, such as news media, public opinion and peers. What about countries with low influence of EBP, what informs policy here? Using data from a large-n survey with Brazilian federal bureaucrats we uncover associations between sources of information and factors shaping their preferences, such as policy work and policy capacities. We find that in a civil law system such as the Brazilian administration homemade sources rule: there is a prevalence of use of government sources, especially among bureaucrats performing analytical and oversight tasks, and those in higher positions. Academic sources are associated with higher analytical capacity (of the individual and organisation), but not with any particular policy sector. By investigating an important yet often neglected issue in EBP – the role of different types of information and how they inform policy – this article contributes to the literatures on policy work and policy capacity, especially given its empirical focus on Brazil.

Non-Empirical Uncertainties in Evidence-Based Decision Making

Abstract The increasing success of the evidence-based policy movement is raising the demand of empirically informed decision making. As arguably any policy decision happens under conditions of uncertainty, following our best available evidence to reduce the uncertainty seems a requirement of good decision making. However, not all the uncertainty faced by decision makers can be resolved by evidence. In this paper, we build on a philosophical analysis of uncertainty to identify the boundaries of scientific advice in policy decision making. We start by introducing a distinction between empirical and non-empirical types of uncertainty, and we explore the role of two non-empirical uncertainties in the context of policy making. We argue that the authority of scientific advisors is limited to empirical uncertainty and cannot extend beyond it. While the appeal of evidence-based policy rests on a view of scientific advice as limited to empirical uncertainty, in practice there is a risk of over reliance on experts beyond the legitimate scope of their authority. We conclude by applying our framework to a real-world case of evidence-based policy, where experts have overstepped their boundaries by ignoring non-empirical types of uncertainty.

Evaluation, Science and Pragmatism

In Aotearoa New Zealand, as elsewhere, the evidence-based policy movement has been one of the most visible recent influences on how policies are described, discussed and debated. It is now commonly taken for granted that good policy work involves using evidence, and that it is important to increase the influence of data and research uptake during policy development. Promoting evidence-based policy has even been used as the raison d’être for the founding of a political party. However, the voices and perspectives of practitioners themselves are often missing from conversations about evidence’s role in policy work. Drawing on my doctoral research, this article presents three stances that frame how policy workers approach evidence in their practice.

A framework for open policy analysis

Abstract The evidence-based policy movement promotes the use of empirical evidence to inform policy decision-making. While several social science disciplines are undergoing a ‘credibility revolution’ focused on openness and replication, policy analysis has yet to systematically embrace transparency and reproducibility. We argue that policy analysis should adopt the open research practices increasingly espoused in related disciplines to advance the credibility of evidence-based policy making. We first discuss the importance of evidence-based policy in an era of increasing disagreement about facts, analysis, and expertise. We present a novel framework for ‘open’ policy analysis (OPA) and how to achieve it, focusing on examples of recent policy analyses that have incorporated open research practices such as transparent reporting, open data, and code sharing. We conclude with recommendations on how key stakeholders in evidence-based policy can make OPA the norm and thus safeguard trust in using empirical evidence to inform important public policy decisions.

The realities of scaling within evidence-based policy

AbstractThe economic model for scaling described by Al-Ubaydli and colleagues offers recommendations to policymakers who make decisions about whether or not to implement evidence-based programs. The core economic model does not currently acknowledge the broader context of policy decision-making and therefore may fail to achieve its objectives. The model focuses primarily on the generation and use of available research in the decision on whether to scale a program. Research studying the use of evidence in policymaking points to a complex set of factors beyond just the strength of the evidence such as leadership, relationships, timing and financial resources that contribute to decisions to scale a program. Second, there is already a robust evidence-based policy movement at the federal, state and local levels. The economic model should leverage this movement rather than providing recommendations that might stall or redirect the movement. The economic model can push the field to strengthen the available evidence while providing recommendations on selecting models to scale within the currently available evidence. This commentary finishes with suggestions for moving forward.

The evidence-based policy movement and political idealism

The opposing views in the scholarly debate on evidence-based policy (EBP) have recently been labeled ‘rationalist’ and ‘constructivist’, where the former are positive to EBP and the latter are not. This framing of the debate is suboptimal, as it conflates critical positions that should be kept separate. This article suggests that the debate should be understood as one between idealists, realists, and counter-idealists about EBP. The realist position, that is, that EBP is difficult or impossible to achieve in practice, has already been treated at length in the debate. The conflict between idealism and counter-idealism, to the contrary, has been neglected. This article aims to stimulate the scholarly debate on EBP by initiating a principled discussion between idealism and counter-idealism about EBP, which should motivate proponents of EBP to formulate their ideal with substantial moral arguments. This places the debate on EBP in the context of normative political theory, where it rightfully belongs.

Tackling the Big Questions in Social Impact Bond Research through Comparative Approaches

This special issue of theJournal of Comparative Policy Analysis brings together four cross-disciplinary articles representing the first concerted attempt to combine comparative approaches to extend theoretical and empirical understandings of Social Impact Bonds (SIBs). SIBs are investment-backed payment-by-results projects and have been subject to vigorous academic debate on their appropriateness and efficacy since the first SIB launched in 2010. This introduction to the special issue outlines the state of the academic literature on SIBs, identifying gaps and suggesting five big questions that do not yet have satisfactory answers: (1) What are the administrative or political problems to which SIBs respond? (2) Where and why do SIBs emerge in particular contexts? (3) What is the role of SIBs in the evidence-based policy movement? (4) Is delivering an intervention through a SIB more effective than other means and are associated costs justifiable? (5) Do SIBs catalyse wider organizational, system, or institutional changes? This introduction then summarizes the articles included in this special issue, discusses how they respond to these big questions, and suggests that further comparative research might best address remaining gaps in the literature.

From the Editorial Board: Using Quantitative Methods to Answer Critical Questions

From the Editorial Board:Using Quantitative Methods to Answer Critical Questions Caitlin E. Kearney On September 12, 2019, during a debate among Democratic presidential hopefuls, Kamala Harris cited a study on teacher-student race matching. If a Black child has one Black teacher by the end of third grade, the study found, they are 13% more likely to go to college; if that child has two Black teachers before the end of third grade, they are 32% more likely to attend college (Gershenson et al, 2018). Harris cited these statistics to highlight the need for investment in Historically Black Colleges and Universities (HBCUs) and their teacher training programs. In an instant, an academic study of Black teachers reached an audience of millions and, in doing so, Harris proved the power of quantitative methods in shaping a policy debate. One measure of the impact of academic research, although imperfect, is the salience of findings in policy debates. In riding the wave of the evidence-based policy movement, rigorous quantitative studies are often prioritized over qualitative studies. In practice, qualitative and quantitative methodologies often work together in a given subfield. Decades of qualitative literature on the impacts of Black teachers—and damages of an overwhelmingly white teaching force on student-level outcomes, particularly for Black children—provided the theoretical framework for Gershenson and colleagues (e.g., Foster, 1990; Irvine, 1989; Kelly, 2010; Ladson-Billings, 2016; Steele, 1997; Tyson, 2003). Yet the conciseness and seeming finality of statistical findings present in quantitative studies often make the findings stick; policymakers are more convinced by compelling numbers than narrative explanations (Branscomb, 2004). Leveraging data from the Tennessee Student/Teacher Achievement Ratio (STAR) class size experiment and state administrative data from North Carolina, Gershenson and colleagues (2018) employed quasi-experimental economic methods to suggest a causal link between exposure to Black teachers and educational attainment for Black students. Their conclusion was striking. The vestiges of the methodological paradigm wars are still very much with us. The goal of this piece is not to resurrect the tiresome battle between quantitative and qualitative methods, but rather to examine the use of quantitative methods to answer critical policy relevant questions. Quantitative Methods and Education Policy Education research investigates a variety of overlapping issues such as teaching and learning, and the political, economic, organizational, and social contexts in which they occur. Coleman (1976) posits that all education research is inherently policy relevant. Others define education policy research more narrowly, arguing that it must analyze policies by defining specific problems, interrogating alternatives, confronting tradeoffs, and utilizing evidence to make a recommendation (e.g., Bardach, 2012). Over the last several decades, there have been growing calls for education research [End Page 195] to be both policy relevant and empirically rigorous (Institute of Education Sciences (IES), 2013; Shavelson & Towne, 2002). Most policy briefs from federal agencies cite quantitative research, and many name particular quantitative methods as the "gold standard" (IES, 2003). Quantitative education researchers have heavily borrowed from economics over the last several decades. Economic analytical frameworks such as the education production function, which draws on human capital theory, predominate in education policy debates. Similarly, numerous quantitative methodologies within education policy research derive from quasi-experimental designs developed by econometricians. Mounting pressure for organizations to be held accountable for their role in societal inequities undergirds much of the aggressive promotion for rigorous research. Landmark legislation, such as the No Child Left Behind Act (2002) and the Education Sciences Reform Act (2002), explicitly endorse scientific research as paths to address intractable social problems. The full history of quantitative methods, however, must be told. There is more to the story than more rigorous methodology; the history of quantitative methods is inextricably intertwined with our nation's past and present culture of white supremacy and cisheteropatriarchy. Early usage of quantitative methods, such as intelligence quotient (IQ) tests and The Bell Curve (Herrnstein & Murray, 1994), constructed and reinforced systems of oppression. The rise of quantitative methods in education is often characterized as a shift toward objectivity and the onset of trustworthy research; however, embracing such methodologies comes with a host of hidden ideological assumptions rendered invisible by claims of neutrality (Davidson, 2018). Others have gone even further...

The role of psychology in evidence-based policymaking: Mapping opportunities for strategic investment in poverty reduction.

Increasing efforts are being undertaken to understand how to improve the use of research evidence in policy settings. In particular, growing efforts to understand the use of research in legislative contexts. Although high-profile examples of psychology's contributions to public policy exist-particularly around antipoverty legislation-little systematic review has quantified how the field has informed federal policy across time. Recognizing the importance of exploring psychology's use in policymaking, we provide an overview of psychology's presence in federal antipoverty legislation over the last 2 decades by reviewing the over 6,000 antipoverty bills introduced to the U.S. Congress since 1993 for mentions of psychology. Further, to explore how psychology's contributions are related to policymakers' attributions about the causes of poverty, their public statements and voting behavior is considered. Key gaps in our scientific knowledge for informing poverty-related policy are identified. Opportunities to enhance the relevance of psychology in poverty reduction efforts, including the evidence-based policy movement, are described. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

From Research Partner to Embedded Criminologist: A Synthesized Taxonomy and Reflections From the Field

Abstract Perhaps never before has the need for collaboration between academic researchers and police agencies been more salient. Technological advances have brought advanced hardware systems into daily policing operations, and analytical software and information systems have assembled massive volumes of data repositories. To produce usable knowledge from these information troves, advanced analytical knowledge has become necessary. At the same time, the evidence-based policy movement requires that police agencies equip themselves with the ability to interpret and produce evaluations of new policing initiatives using sound scientific evaluation methods, something which also requires specialized and advanced knowledge not usually housed within policing organizations. In response to these demands, researcher–police collaborations have become more prevalent. Yet, the specific nature of these researcher–police relationships can take many forms. This article synthesizes previous literature and develops a taxonomy for classifying researcher–police partnerships and discusses the experiences of moving from an external research partner to an embedded criminological ‘unit’ within the Miami Police Department.

When Evidence is not Taken for Granted: The Use and Perception of “Evidence” in the Czech Republic Ministries

Abstract The role of evidence in policy-making is one of the most researched topics in public policy and public administration. However, surprisingly little research has been done on how public officials actually use evidence in everyday life practice. Moreover, these studies have been limited to countries that have been influenced by the evidence-based policy movement (EBP). Little is known about how the evidence is conceptualized and utilized in other countries which have not been so strongly influenced by EBP movement. This paper addresses this gap. Using a large-N survey on the Czech ministerial officials and in-depth interviews with them, we explore what is understood under the term of “evidence”, what kind of evidence is used and preferred by public officials and why. In doing so, we use four theoretical perspectives on the use of evidence. We show that despite the long-established tradition of using research in policy-making the importance of research evidence in the Czech Republic is far from being taken for granted. On the contrary, the immediate and personal experience is often preferred over the research findings. The exception to that are census-like statistical data and comparative data published by international organizations. We find some support for the two-communities metaphor, though these communities are not defined by their socio-demographic characteristics, but rather by their internal discourse and understanding of evidence.

The Roles Foundations Are Playing in the Evidence-Based Policy Movement

Foundations are actively engaged in supporting evidence-based policymaking through collaborative funding, supporting, and creating intermediary organizations; building the infrastructure needed to support evidence-based policymaking; and improving the relevance of research for practice and policy. For a variety of reasons, they are shifting from a focus on the federal government and the identification of effective brand name innovations to an emphasis on supporting local actors to design and test solutions using local data. This article provides examples of foundation work, describes and discusses how and why it is evolving, and uses historical examples to place the change in context.

The Role of Evaluation in Building Evidence-Based Policy

Growing recognition that many government programs may be ineffective, or at best of unproven effectiveness, has led to the evidence-based policy movement—an effort to ensure that proposed and existing public programs have been shown to achieve their objectives. Rigorous evaluation is central to this movement. In this article, I briefly review the history of evaluation of social programs and the barriers to the application of effective evaluation to public policy.

OP119 Appraising Qualitative Research For Qualitative Evidence Syntheses

Introduction:The growth of the evidence-based policy movement sought to determine how to better assess and incorporate qualitative evidence in clinical practice and policy development. The question engendered was not whether qualitative research is valuable but how researchers can enhance its rigor. From this discussion arose over one hundred appraisal tools for the quality appraisal process of qualitative studies. For those without a deep familiarity with the qualitative research paradigm, navigating through the breadth of tools to find the most suitable tool for the task is a cumbersome process. This presentation will review the descriptive characteristics of available quality appraisal tools for assessing the quality of primary qualitative studies in qualitative evidence syntheses (QES). This presentation will also offer a critical discussion on the use of reflexivity as a de facto quality criterion, and how methodological reporting may influence the application of quality criteria in QES.Methods:We conducted a systematic search to identify quality appraisal tools of qualitative research designed for use in QES. This search built upon the work of Santiago-Delefosse and colleagues by extending their search to 2016.Results:We identified eight appraisal tools intended for use in the quality appraisal process of a QES. We provide a description of the structure, content, objectives, and philosophies of tools followed by considerations concerning their historical antecedents, common patterns regarding structure, content, and purpose, and the implications of these patterns on the QES process.Conclusions:Quality appraisal of qualitative research is an important step in QES, and there have been a proliferation of tools for this purpose. By providing an overview of available tools detailing their intent and strengths, this presentation will assist those engaging in QES to choose an appropriate tool for their work.

The many meanings of evidence: a comparative analysis of the forms and roles of evidence within three health policy processes in Cambodia

BackgroundDiscussions within the health community routinely emphasise the importance of evidence in informing policy formulation and implementation. Much of the support for the evidence-based policy movement draws from concern that policy decisions are often based on inadequate engagement with high-quality evidence. In many such discussions, evidence is treated as differing only in quality, and assumed to improve decisions if it can only be used more. In contrast, political science scholars have described this as an overly simplistic view of the policy-making process, noting that research ‘use’ can mean a variety of things and relies on nuanced aspects of political systems. An approach more in recognition of how policy-making systems operate in practice can be to consider how institutions and ideas influence which pieces of evidence appear to be relevant for, and are used within, different policy processes.MethodsDrawing on in-depth interviews undertaken in 2015–2016 with key health sector stakeholders in Cambodia, we investigate the evidence perceived to be relevant to policy decisions for three contrasting health policy examples, namely tobacco control, HIV/AIDS and performance-based salary incentives. These cases allow us to examine the ways that policy-relevant evidence may differ given the framing of the issue and the broader institutional context in which evidence is considered.ResultsThe three health issues show few similarities in how pieces of evidence were used in various aspects of policy-making, despite all being discussed within a broad policy environment in which evidence-based policy-making is rhetorically championed. Instead, we find that evidence use can be better understood by mapping how these health policy issues differ in terms of the issue characteristics, and also in terms of the stakeholders structurally established as having a dominant influence for each issue. Both of these have important implications for evidence use. Contrasting concerns of key stakeholders meant that evidence related to differing issues could be understood in terms of how it was relevant to policy. The stakeholders involved, however, could further be seen to possess differing logics about how to go about achieving their various outcomes – logics that could further help explain the differences seen in evidence utilisation.ConclusionA comparative approach reiterates that evidence is not a uniform concept for which more is obviously better, but rather illustrates how different constructions and pieces of evidence become relevant in relation to the features of specific health policy decisions. An institutional approach that considers the structural position of stakeholders with differing core goals or objectives, as well as their logics related to evidence utilisation, can further help to understand some of the complexities of evidence use in health policy-making.

Policy capacity and evidence-based policy in the public service

ABSTRACTGovernments in many jurisdictions have called for an increase in ‘evidence-based’ policy-making. However, the international evidence-based policy movement has so far shown little progress in transforming the way that public policy is formulated and implemented. Much research on evidence-based policy has focused on political interference and contextual frames of reference as barriers to the uptake of research evidence. With the support of data from a survey of over 2,000 Australian public servants, we argue that individual and organizational deficits can leave the public service structurally unprepared for an engagement with diverse forms of evidence, including academic research in particular.

Agenda Setting and Evidence in Maternal Health: Connecting Research and Policy in Timor-Leste.

The evidence-based policy (EBP) movement has received significant attention in the scientific literature; however, there is still very little empirical research to provide insight into how policy decisions are made and how evidence is used. The lack of research on this topic in low- and middle-income countries is of particular note. We examine the maternity waiting home policy in Timor-Leste to understand the role of context, policy characteristics, individual actors, and how evidence is used to influence the policy agenda. The research tracked the maternity waiting home policy from 2005 to 2009 and is based on in-depth interviews with 31 senior policy-makers, department managers, non-government organization representatives, and United Nations advisors. It is also informed by direct observation, attendance at meetings and workshops, and analysis of policy documents. The findings from this ethnographic case study demonstrate that although the post-conflict context opened up space for new policy ideas senior Ministry of Health officials rather than donors had the most power in setting the policy agenda. Maternity waiting homes were appealing because they were a visible, non-controversial, and logical solution to the problem of accessing maternal health services. Evidence was used in a variety of ways, from supporting pre-determined agendas to informing new policy directions. In the pursuit of EBP, we conclude that the power of research to inform policy lies in its timeliness and relevance, and is facilitated by the connection between researchers and policy-makers.

Revisiting the “two communities” metaphor of research utilisation

Purpose– The “two communities” metaphor for the relationship between policy and academia is inconsistent with empirical evidence that shows that a sizeable minority of public servants use academic research in their policy-related work. The purpose of this paper is to investigate the two communities metaphor by comparing the personal and professional characteristics of Australian public servants who claim to use research in their policy work with the characteristics of those who claim not to use research.Design/methodology/approach– Using data from a survey of 2,084 public servants from the state and federal levels in Australia carried out from 2011 to 2013, tests of statistical significance were conducted for the relationship between some personal and professional characteristics (e.g. gender, age, work experience) and the claim that research is used in policy-related work.Findings– The “two communities” metaphor is not an accurate description of the relationship between policy and academia. In reality, public servants who claim to use academic research in their policy work are more likely to have much in common with academics, including having postgraduate degrees and work experience in the university sector.Research limitations/implications– Rather than existing as isolated solitudes, the findings in this paper suggest that the policy and academic communities possess links that can encourage the use of research in policy making.Originality/value– The findings presented in this paper are especially important for the evidence-based policy movement, which emphasises the value of the use of research evidence in the creation of public policy.

Evidence-informed health policy - the crucial role of advocacy

Health policy makers and clinicians often face similar decision-making challenges. The issues are turbulent, characterised by high risk and complexity, often involve value conflicts and occur in settings of rapid change. Policy makers' decisions are under increasing scrutiny for their use of evidence, with many health policies reflecting political influence rather than rigorous analysis. The evidence-based policy movement offers a range of accounts for this. We argue that advocacy in three critical areas helps explain when evidence is used in the policy making process and then contrast the impacts of advocacy for evidence use in two nutrition policy cases.