Abstract
As policy makers require more rigorous assessments for the strength of evidence in Theory-Based evaluations, Bayesian logic is attracting increasing interest; however, the estimation of probabilities that this logic (almost) inevitably requires presents challenges. Probabilities can be estimated on the basis of empirical frequencies, but such data are often unavailable for most mechanisms that are objects of evaluation. Subjective probability elicitation techniques are well established in other fields and potentially applicable, but they present potential challenges and might not always be feasible. We introduce the community to a third way: simulated probabilities. We provide proof of concept that simulation can be used to estimate probabilities in diagnostic evaluation and illustrate our case with an application to health policy.
Highlights
Trusting evaluation evidence is crucial if policy decisions are to be grounded on evaluation findings; whether the latter are credible or not has very practical implications for the decisionmaking process
This paper proposes as a ‘third way’: the use of simulation methods to generate in silico frequencies for particular pieces of evidence in Diagnostic Evaluation, or Bayesian Updating applied to Theory-Based Evaluation
We explore its potential in relation to Theory-Based Evaluation (TBE) as a whole, rather than to particular TBE formulations like Process Tracing or contribution analysis (CA)
Summary
Trusting evaluation evidence is crucial if policy decisions are to be grounded on evaluation findings; whether the latter are credible or not has very practical implications for the decisionmaking process. While policy makers and analysts are developing an increasing interest in Theory-Based Evaluation, the discussion around what constitutes credible evidence in this. Establishing causal connections is often challenging in social systems, due to difficulties in disentangling interdependencies, and in isolating causes from particular contexts and confounding factors (Byrne and Callaghan, 2013). This is exacerbated when time is one of these factors because of the additional difficulty in capturing changing systems and their oftenemergent dynamics. This is a problem for evaluation, in particular impact evaluation, as it is essential to show that a particular (set of) intervention(s) has brought about, or caused, a particular (set of) outcomes
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
