A Beginner’s Guide to Randomized Evaluations in Development Economics

Abstract

I. IntroductionRandomized evaluations in development economics are receiving considerable attention; there has been a dramatic increase in randomized evaluations carried out not only by academia but also by governments, international agencies, and non-governmental organizations to test the effectiveness of a specific program or policy (Banerjee, and Duflo 2009).1 The rise in randomized evaluations is in line with the credibility revolution in empirical economics which emphasizes the identification of causal effects (Angrist, and Pischke 2010). Previously, sensitivity analysis, where researchers presented the rigorousness of their results with different specifications or functional forms had been considered a salutary econometric practice (Leamer 1985). However, the sensitivity analysis was limited in identifying the causal effect of a program because it is difficult to know the difference between the outcome a participant experienced under the program and the experienced outcome if the participant did not take part in the program. Although the causal effect of a program for each person is not identifiable (this is the fundamental problem of causal inference), a randomized evaluation makes it possible to create grouplevel counterfactual which is free from selection bias, and to provide internally valid estimates of the average treatment effect (Rubin 1974).In this paper we provide a brief overview of the important steps in planning and implementing a randomized evaluation from start to finish including issues such as research question development, local partner institution, IRB (Institutional Review Board), PAP (Pre-Analysis Plan), sample size calculation, random sampling, randomization, survey implementation, data entry, and data analysis.2II. Research Question DevelopmentNot all research questions need a randomized evaluation to be answered, thus, it is important to ask the right research question that requires a randomized evaluation. If we are interested to know what the needs are in a specific context, then descriptive need assessment can be enough. However, if we want to know whether a program or policy works or not, then we need to conduct a randomized evaluation to answer this question. Furthermore, randomized evaluations should be performed if we aim to answer one of the following research questions. 1) Which elements of the program matter the most? 2) Which of the two different programs produce a better outcome? 3) Are there complementarities among the programs? 4) Can the results from one context be replicated in another context?A good place to start developing research questions is with existing literature reviews. We can quickly summarize the information in existing research regarding issues of interest. The information can also help identify the most important knowledge gaps, where new research questions begin. High quality literature review on randomized evaluations can be found within institutions such as Jameel Poverty Action Lab (J-PAL) at MIT, Innovations for Poverty Action (IPA), the International Initiative for Impact Evaluation (3ie), and the Development Impact Evaluation Initiative (DIME) in the Research Group of the World Bank.3 Many academic journals in economics, which are all reliable resources, also publish up-to-date literature reviews. These journals include The Handbook of Economics series, the Annual Review of Economics, the Journal of Economic Perspectives, and the Journal of Economic Literature.After developing research questions for a randomized evaluation, the next step is to incorporate the questions into the survey questionnaire to analyze the research hypotheses and its possible mechanisms. It is useful to review the existing survey questionnaire of related research, which is often available on its author's personal webpage or on the publication journal's website.4 Another good source for questionnaire development is the government's or multilateral organization's survey questionnaire such as Demographic and Health Surveys (DHS). …