RealWorld Evaluation: Working Under Budget, Time, Data, and Political Constraints

Abstract

Introduction PART I. OVERVIEW: REALWORLD EVALUATION Chapter 1: RealWorld Evaluation and the Contexts in Which It Is Used Welcome to RealWorld Evaluation The RealWorld Evaluation Context The Four Types of Constraints Addressed by the RealWorld Approach The RealWorld Evaluation Approach to Evaluation Challenges Comparing the RealWorld Evaluation Context and Issues in Developing and Developed Countries Who Uses RealWorld Evaluation, for What Purposes, and When? Summary Further Reading PART II. THE SEVEN STEPS OF THE REALWORLD EVALUATION APPROACH Chapter 2: First clarify the purpose: Scoping the evaluation Stakeholder expectations of impact evaluations Understanding information needs Developing the program theory model Identifying the constraints to be addressed by the RWE and determining the appropriate evaluation design Developing the Terms of Reference for the Evaluation Summary Further reading Chapter 3: Not enough money: Addressing budget constraints Simplifying the evaluation design Clarifying client information needs Using existing data Reducing costs by reducing sample size Reducing costs of data collection and analysis Common threats to validity of budget constraints Summary Further reading Chapter 4: Not enough time: Addressing scheduling and other time constraints Similarities and differences between time and budget constraints Simplifying the evaluation design Clarifying client information needs Using existing documentary data Reducing sample size Rapid data collection methods Reducing time pressures on outside consultants Hiring more resource people Building outcome indicators into project records Data collection and analysis technology Common threats to adequacy and validity relating to time constraints Summary Further reading Chapter 5: Critical information is missing or difficult to collect: Addressing data constraints Data issues facing RealWorld evaluators Reconstructing baseline data Special issues in reconstructing comparison groups Collecting data on sensitive topics or from groups who are difficult to reach Common threats to adequacy and validity relating to data constraints Summary Further reading Chapter 6: Reconciling different priorities and perspectives: Addressing political influences Values, ethics, and politics Political issues at the outset of an evaluation Political issues during the conduct of an evaluation Political issues in evaluation reporting and use RealWorld strategies for addressing political constraints Summary Further reading Chapter 7: Strengthening the evaluation design and the validity of the conclusions Validity in evaluation Factors affecting adequacy and validity Assessing the adequacy of quantitative evaluation designs Strengthening validity in quantitative evaluations Assessing the adequacy of qualitative evaluation designs Strengthening validity in qualitative evaluations Points during the RWE evaluation cycle when corrective measures can be taken Summary Further reading Chapter 8: Making it useful: Helping clients and other stakeholders utilize the evaluation The underutilization of evaluations studies The importance of the RealWorld evaluation scoping phase for utilization Formative evaluation strategies Communicating with clients throughout the evaluation Evaluation capacity building Communicating findings Developing a follow-up action plan Summary Further reading PART III A REVIEW OF EVALUATION METHODS AND APPROACHES AND THEIR APPLICATIONS IN REALWORLD EVALUATION Chapter 9 Applications of program theory in RealWorld evaluation Defining Program Theory Evaluation Applications of Program Theory in evaluation Constructing Program Theory Models Logical framework analysis and results chains Program theory evaluation and causality Summary Further reading Chapter 10: The Most Widely-Used RealWorld Quantitative Evaluation Designs Randomized and quasi-experimental evaluation designs The most widely used quantiative designs Ways to strengthen quantitative RWE designs Seven quasi-experimental designs that cover most RWE scenarios Summary Further reading Chapter 11 Quantitative evaluation methods The quantitative and qualitative traditions in evaluation research Quantitative methodologies Applications of quantitative methodologies in program evaluation Quantitative methods for data collection The management of data collection for quantitative studies Data analysis Summary Further reading Chapter 12 Qualitative evaluation methods Qualitative methodology and tradition Qualitative methodology: An overview Different reasons for using different methodologies Qualitative data collection Qualitative data analysis Summary Further reading Chapter 13 Mixed-method evaluation The mixed method approach Mixed-method strategies Implementing a mixed-method design Summary Further reading Chapter 14 Sampling for RealWorld Evaluation The Importance Of Sampling For RWE Purposive sampling Probability (random) sampling Using power analysis and effect size for estimating the appropriate sample size Determining The Size Of The Sample The Contribution of Meta-Analysis Sampling Issues For Mixed-Method Evaluations Sampling issues for RWE Summary Further reading PART IV. PULLING IT ALL TOGETHER Chapter 15 Learning together: Building Capacity For RealWorld Evaluation Defining evaluation capacity building RealWorld evaluation capacity building Designing and delivering evaluation capacity building Summary Further reading Chapter 16: Bringing it all together: Applying RealWorld evaluation approaches to each stage of the evaluation process Scoping the evaluation Choosing the best design from the available options Determining appropriate methodologies Ways to strengthen RWE designs Staffing the evaluation economically Collect data efficiently Analyze the data efficiently Report findings efficiently and effectively Help clients use the findings well Appendices Glossary of terms References