Impact Evaluation Design Research Articles

The Impact of Strengths-Based Working on Long-Term Housing Outcomes: A Qualitative Comparative Analysis

Abstract Strengths-based working (SBW) is being adopted across multiple sectors, but the evidence base for its effectiveness is limited. In this assessment of a social outcomes contract, authors evaluated the impact of SBW on the delivery of homelessness-related outcomes for a high-risk client group. The novel impact evaluation design first identified characteristics in frontline staff associated with SBW and then decoded conjectural causal links between these characteristics and their clients’ social outcomes using qualitative comparative analysis. Data were collected from 65 frontline workers, measuring reflective practice, self-efficacy, and innovativeness. Authors found that clients of staff who most strongly exhibited characteristics associated with SBW achieved better long-term outcomes. This article makes four important contributions to our understanding of SBW. First, authors show how it is possible to undertake rigorous impact evaluation of SBW in a complex setting and identify moderately complex causal relations with limited sample size. Second, authors propose and test the presence of key characteristics associated with SBW and, in doing so, contribute to the theoretical understanding of SBW. Third, authors present new evidence on the effectiveness of SBW. Finally, the article demonstrates how social outcomes contracts can provide a framework for testing innovative social interventions.

Spatiotemporal Land-Use Dynamics in Continental Portugal 1995–2018

Monitoring land-use patterns and its trends provides useful information for impact evaluation and policy design. The latest in-depth studies of land-use dynamics for continental Portugal are outdated, and have not examined how municipalities may be classified into a typology of observed dynamics or considered the trajectory profiles of land-use transitions. This paper presents a comprehensive analysis of the spatiotemporal dynamics of land-use in continental Portugal from 1995 to 2018. Our multi-scalar approach used land-use maps in geographic information systems with the following objectives: (i) quantify variations of land-use classes, (ii) assess the transitions between uses, and (iii) derive a municipal typology of land-use dynamics. The methodology employed involved calculating statistical indicators of land-use classes, transition matrices between uses and combinatorial analysis for the most common trajectory-profiles. For the typology, a principal component analysis was used for dimensionality reduction and the respective components were classified by testing several clustering techniques. Results showed that the land-use transitions were not homogeneous in space or time, leading to the growth of territorial asymmetries. Forest (Δ5%), water bodies (Δ28%) and artificial surfaces (Δ35%) had a greater expansion, as opposed to agricultural areas, which had the biggest decline (Δ-8%). Despite the decline of agricultural activities, olive-grove expansion (Δ7%) was a relevant dynamic, and in the case of forests, the increment of eucalyptus (Δ34%) replaced native species such as the maritime pine (Δ-20%). A land-use-dynamics typology was estimated, dividing continental Portugal into 11 clusters, which is informative for sectoral policies and spatial planning, as zonings in need of interventions tailored to their specificities. The findings are a contribution to the study of land-use dynamics in continental Portugal, presenting various challenges for sustainable land uses with regard to the urban system, forest management, food production, soil preservation, and ecosystem protection.

Using simulation modelling and systems science to help contain COVID-19: A systematic review.

This study systematically reviews applications of three simulation approaches, that is, system dynamics model (SDM), agent‐based model (ABM) and discrete event simulation (DES), and their hybrids in COVID‐19 research and identifies theoretical and application innovations in public health. Among the 372 eligible papers, 72 focused on COVID‐19 transmission dynamics, 204 evaluated both pharmaceutical and non‐pharmaceutical interventions, 29 focused on the prediction of the pandemic and 67 investigated the impacts of COVID‐19. ABM was used in 275 papers, followed by 54 SDM papers, 32 DES papers and 11 hybrid model papers. Evaluation and design of intervention scenarios are the most widely addressed area accounting for 55% of the four main categories, that is, the transmission of COVID‐19, prediction of the pandemic, evaluation and design of intervention scenarios and societal impact assessment. The complexities in impact evaluation and intervention design demand hybrid simulation models that can simultaneously capture micro and macro aspects of the socio‐economic systems involved.

Assessing the effect of an educational intervention on early childhood development among Mexican preschool children in the state of Oaxaca: a study protocol of a cluster randomized stepped-wedge trial

BackgroundEarly childhood development (ECD) is essential in human capacity building and a critical element in the intergenerational process of human development. In some countries, social programs targeted at improving ECD have proven to be successful. Oaxaca is one of the States with the greatest social inequities in Mexico. Therefore, children in Oaxaca are at a high risk of suboptimal ECD. In 2014, the non-governmental organization (NGO) Un Kilo de Ayuda started to implement the Neurological and Psycho-affective Early Childhood Development Program in eighty marginalized communities of Oaxaca. In this article, we present the impact evaluation design to estimate the effect of this program on ECD.MethodsWe will use a cluster randomized stepped-wedge design with an allocation ratio of 1:1. Communities will be randomly assigned to each study group: four groups of twenty communities each. We expect that children from intervened communities will show better ECD outcomes.DiscussionThis study is one of the few rigorous assessments of the effect of an ECD program on the neurodevelopment of Mexican children recruited in their first 3 years of life from communities of high social vulnerability. Our study design is recommended when the way in which outcomes are measured and assessed depends on age, self-selection is present, and assignment is performed at an aggregate level. Implementation research will be conducted prior to study launch and quality control measures will be in place to maximize the fidelity of study design implementation.Trial registrationClinicalTrials.gov NCT04210362

Assessing the Contribution to Market System Change of the Private Enterprise Programme Ethiopia

An impact evaluation of a pro-poor market system development programme, ‘Making Markets Work for the Poor’ (M4P), poses several methodological issues for evaluators. M4P interventions intend to change the contextual conditions in which stakeholders take business decisions so that it triggers change processes in the wider social system and ultimately benefits poor people. An impact evaluation design for such a programme thus needs to be robust enough to adequately capture these systemic outcomes, acknowledging dynamic changes in intervention delivery as well as in market conditions over time. Theory-based evaluation can provide learning and accountability when it incorporates methods that allow a critical reflection on the key causal steps in an intervention’s theory of change. We present our learnings about indicators and methods to reflect on the importance of the contributions to market system change of a large M4P programme in Ethiopia.

Evaluating impact from research: A methodological framework

BackgroundInterest in impact evaluation has grown rapidly as research funders increasingly demand evidence that their investments lead to public benefits. AimsThis paper analyses literature to provide a new definition of research impact and impact evaluation, develops a typology of research impact evaluation designs, and proposes a methodological framework to guide evaluations of the significance and reach of impact that can be attributed to research. MethodAn adapted Grounded Theory Analysis of research impact evaluation frameworks drawn from cross-disciplinary peer-reviewed and grey literature. ResultsRecognizing the subjective nature of impacts as they are perceived by different groups in different times, places and cultures, we define research impact evaluation as the process of assessing the significance and reach of both positive and negative effects of research.Five types of impact evaluation design are identified encompassing a range of evaluation methods and approaches: i) experimental and statistical methods; ii) textual, oral and arts-based methods; iii) systems analysis methods; iv) indicator-based approaches; and v) evidence synthesis approaches.Our guidance enables impact evaluation design to be tailored to the aims and context of the evaluation, for example choosing a design to establish a body of research as a necessary (e.g. a significant contributing factor amongst many) or sufficient (e.g. sole, direct) cause of impact, and choosing the most appropriate evaluation design for the type of impact being evaluated. ConclusionUsing the proposed definitions, typology and methodological framework, researchers, funders and other stakeholders working across multiple disciplines can select a suitable evaluation design and methods to evidence the impact of research from any discipline.

Build an Artificial Coral Reef Community Extension Project: An Impact Study

Coral reefs are ecologically important as they provide shelter and life support to the marine resources, sustain ecological balance, and human existence. Coral reef degradation is an ecological problem that needs human intervention and is considered a school’s social responsibility. The USANT College of Maritime Education and College of Engineering and Architecture rationalized social responsibility in partnership with the Bureau of Fisheries and Aquatic Resources (BFAR) and the Philippine Coast Guard Auxiliary (PCGA) and the barangay Pararao, Balatan, Camarines Sur, Region V, Philippines. The researcher used the Impact Evaluation and Study Design substantiated by an FGD and Underwater Diving of BFAR marine specialists to ascertain the impacts of the artificial coral reef project in drawing environmental awareness and determine the actual effect of the artificial coral reefs as shelters and spawning sites of fishes. Findings reveal that communal participation in safeguarding the vicinity of the reefs emanating at the barangay level is evident; the community folks had gained an understanding of the role of coral reefs in the ecosystem and school of fishes shoaled in the vicinity. Thus, this coral reef project drew positive attention, reduced minimalistic attitudes, and increased environmental awareness. The artificial reefs now serve as natural habitats. Active partnership and participation of the stakeholders and the provision of budget are imperative to sustainable project implementation.

WITHDRAWN: Evaluating impact from research: A methodological framework

Abstract Background Interest in impact evaluation has grown rapidly as research funders increasingly demand evidence that their investments lead to public benefits. Aims This paper analyses literature to provide a new definition of research impact and impact evaluation, develops a typology of research impact evaluation designs, and proposes a methodological framework to guide evaluations of the significance and reach of impact that can be attributed to research. Method An adapted Grounded Theory Analysis of research impact evaluation frameworks drawn from cross-disciplinary peer-reviewed and grey literature. Results • Recognizing the subjective nature of impacts as they are perceived by different groups in different times, places and cultures, we define research impact evaluation as the process of assessing the significance and reach of both positive and negative effects of research. • Five types of impact evaluation design are identified encompassing a range of evaluation methods and approaches: i) experimental and statistical methods; ii) textual, oral and arts-based methods; iii) systems analysis methods; iv) indicator-based approaches; and v) evidence synthesis approaches. • Our guidance enables impact evaluation design to be tailored to the aims and context of the evaluation, for example choosing a design to establish a body of research as a necessary (e.g. a significant contributing factor among many) or sufficient (e.g. sole, direct) cause of impact, and choosing the most appropriate evaluation design for the type of impact being evaluated. Conclusion Using the proposed definitions, typology and methodological framework, researchers, funders and other stakeholders working across multiple disciplines can select a suitable evaluation design and methods to evidence the impact of research from any discipline.

Assessing the downstream socioeconomic impacts of agroforestry in Kenya

Abstract Agroforestry is widely purported to improve the livelihoods of smallholder farmers, rehabilitate degraded landscapes, and enhance the provisioning of ecosystem services. Yet, evidence supporting these longer-term impacts is limited. Using a quasi-experimental impact evaluation design informed by a theory-based and mixed methods framework, we investigated selected intermediate and final outcomes of a nine-year effort led by Vi Agroforestry, a Swedish non-governmental organization (NGO), to promote agroforestry in large sections of Bungoma and Kakamega counties in western Kenya. We compared households belonging to 432 pre-existing farmer groups operating in 60 program villages and 61 matched comparison villages. To address potential self-selection bias, we used program targeting as an instrument for program participation, combined with the difference-in-differences approach to control for time-invariant differences between our treatment and comparison groups. We complemented the above with semi-structured interviews with a sub-sample of 40 purposively selected program participants. Despite evidence of variable program exposure and agroforestry uptake, we found modest, yet statistically significant, effects of Vi Agroforestry’s program on intermediate outcomes, such as agroforestry product income, fuelwood access, and milk yields among dairy farmers. We also found that this program modestly increased asset holdings, particularly among households represented by female program participants.

Corrigendum: Spirituality and impact evaluation design: The case of an addiction recovery faith-based organisation in Argentina

No abstract available.

Prospects for Agricultural Sustainable Intensification: A Review of Research

In recent years, as a way to achieve higher agricultural output while reducing the negative impact of agricultural production on the environment, agricultural sustainable intensification has attracted worldwide attention. Under the framework of "connotation definition-measuring method-influencing factor-implementation path", this paper systematically sorts out the main research results in the field of agricultural sustainable intensification. The results show that: (1) The connotation of agricultural sustainable intensification has not been clearly defined. It is widely believed that sustainable intensification has the characteristics of increasing production and reducing environmental damage, and is widely used in agricultural, biological and environmental sciences; (2) The measurement methods and indicators of agricultural sustainable intensification are diverse, and the measurement cases are mainly distributed in Europe, Asia, Africa and America; (3) The influencing factors of agricultural sustainable intensification can be roughly divided into four aspects: socio-economic factors, farmers’ own characteristics and natural factors, among which population pressure is the potential driving force for agricultural sustainable intensification; (4) The most obvious feature of agricultural sustainable intensification is the reduction of the yield gap. The strategy of implementing agricultural sustainable intensification can be attributed to the effective use of inputs and the adoption of sustainable practices and technologies. Therefore, the implementation path can be summarized as enhancing the effectiveness of external inputs to the agricultural system and optimizing the practice and technology mix within the crop production system. Finally, this paper concludes that research on connotation definition, influencing mechanism, different regional models, incentive mechanism for farmers, impact evaluation and system design of agricultural sustainable intensification should be strengthened in future.

Spirituality and impact evaluation design: The case of an addiction recovery faith-based organisation in Argentina

The importance of the spiritual dimension in the lives of people living in conditions of poverty and social exclusion and the often-critical role of faith-based organisations has gained increasing relevance in development research and practice. A growing line of research focuses on how to integrate the faith dimension into the evaluation of social programmes and on quantifying the effects of faith. The objective of this article is to propose a framework for integrating a spiritual dimension into the design and practice of impact evaluation by using the concept of integral human development (IHD). Integral human development emerged within the Catholic social tradition, but is not specific to it. It is a perspective on human development that integrates the material and spiritual, recognises the interdependence between humans and their environment or territory and demands change at both the individual and collective levels. This framework is then applied to the design of an impact evaluation of a faith-based programme that accompanies people with drug and alcohol addictions in Argentina. The article highlights the following characteristics of an integral impact evaluation: the interaction between multiple well-being dimensions; the use of knowledge and methods of analysis from multiple disciplines; the importance of understanding the diverse pathways to improvements in well-being; the assessment of the spiritual dimension through changes in one’s relationships with oneself, others and the environment; and the importance of assessing personal change within the context of social and community transformation.

An Analysis on Rural Development Through MGNREGS in Balangir District of Odisha

Mahatma Gandhi National Rural Employment Guarantee Scheme (MGNREGS) played a very vital role by providing wage employment for one hundred days in a year to each rural household whose adult member volunteer to do unskilled manual work. It acts as a fall back measures during these lean seasons by becoming an alternative source of income for these poor household, acting as a social safety net for them. This study analyses the impact of the program on the rural development . Rural development is the overall development of rural areas with respect to food security, economic security, habitat security, health security, nutritional security etc. So, respective index are being constructed to find the impact on rural development. Randomized selection of beneficiary and non-beneficiary communities has been done. A randomisation impact evaluation design was used with households eligible and the rural development index (RDI) is constructed. It has been found that there is significant impact of the program on the rural development.

Methodological Considerations for Use of Routine Health Information System Data to Evaluate Malaria Program Impact in an Era of Declining Malaria Transmission.

.Coverage of malaria control interventions is increasing dramatically across endemic countries. Evaluating the impact of malaria control programs and specific interventions on health indicators is essential to enable countries to select the most effective and appropriate combination of tools to accelerate progress or proceed toward malaria elimination. When key malaria interventions have been proven effective under controlled settings, further evaluations of the impact of the intervention using randomized approaches may not be appropriate or ethical. Alternatives to randomized controlled trials are therefore required for rigorous evaluation under conditions of routine program delivery. Routine health management information system (HMIS) data are a potentially rich source of data for impact evaluation, but have been underused in impact evaluation due to concerns over internal validity, completeness, and potential bias in estimates of program or intervention impact. A range of methodologies were identified that have been used for impact evaluations with malaria outcome indicators generated from HMIS data. Methods used to maximize internal validity of HMIS data are presented, together with recommendations on reducing bias in impact estimates. Interrupted time series and dose-response analyses are proposed as the strongest quasi-experimental impact evaluation designs for analysis of malaria outcome indicators from routine HMIS data. Interrupted time series analysis compares the outcome trend and level before and after the introduction of an intervention, set of interventions or program. The dose-response national platform approach explores associations between intervention coverage or program intensity and the outcome at a subnational (district or health facility catchment) level.

A meta-framework for conducting mixed methods impact evaluations: Implications for altering practice and the teaching of evaluation

Educational interventions are becoming increasingly more complex, far-reaching, and high-stakes. Thus, there is a need for an evaluation meta-framework that is comprehensive, flexible, and meets enhanced complexity. Therefore, we provide a new and comprehensive definition of impact evaluations—what we call a comprehensive impact evaluation—that draws out the importance of collecting and analyzing both quantitative and qualitative data, thereby resulting in a rigorous approach that can allow for strong inferences. Further, we provide an overview of impact evaluation designs that can be used in comprehensive impact evaluations along with a rationale for using both quantitative and qualitative research techniques. This leads to the central purpose of the article: to provide a meta-framework for conducting what we call a Mixed Methods Theory-Based Impact Evaluations, wherein mixed methods techniques are used at every phase of the process. Building on White’s (2009) work, we outline an 8-phase Mixed Methods Theory-Based Impact Evaluation.

Impact Evaluation Design for Community Midwife Technicians in Malawi

Maternal Mortality Ratio and neonatal mortality rate are alarmingly high in Malawi. The shortage and poor retention of midwives coupled with poor working conditions have been a major challenge affecting the provision of high-quality maternity care for women. Many women are giving birth without skilled attendants, increasing the risk of maternal and neonatal illness and death. The major driving factor in the shortage of health staff is the limited number of existing training slots and hence the minimum output from the training institutions into service delivery units. Midwifery is a key component of sexual, reproductive, maternal, and newborn healthcare. Responding to the crisis, the Malawi Government has made a commitment in strengthening human resources for health, including accelerating training and recruitment of health professionals to fill all the shortage gaps in the health sector. One mechanism implemented by Malawi Government to increase skilled attendance at birth in rural areas is the introduction of Community midwifery assistants (CMA) training. This program of training community midwives is being piloted and targets the general population of pregnant women and their new born babies in rural areas where the CMAs are deployed. However, there is a great need to have this initiative evaluated and gauge its impact in attaining the desired outcomes.

Measuring and understanding the effects of a performance based financing scheme applied to nutrition services in Burundi-a mixed method impact evaluation design.

BackgroundMalnutrition is a huge problem in Burundi. In order to improve the provision of services at hospital, health centre and community levels, the Ministry of Health is piloting the introduction of malnutrition prevention and care indicators within its performance based financing (PBF) scheme. Paying for units of services and for qualitative indicators is expected to enhance provision and quality of these nutrition services, as PBF has done, in Burundi and elsewhere, for several other services.MethodsThis paper presents the protocol for the impact evaluation of the PBF scheme applied to malnutrition. The research design consists in a mixed methods model adopting a sequential explanatory design. The quantitative component is a cluster-randomized controlled evaluation design: among the 90 health centres selected for the study, half receive payment related to their results in malnutrition activities, while the other half get a budget allocation. Qualitative research will be carried out both during the intervention period and at the end of the quantitative evaluation. Data are collected from 1) baseline and follow-up surveys of 90 health centres and 6,480 households with children aged 6 to 23 months, 2) logbooks filled in weekly in health centres, and 3) in-depth interviews and focus group discussions. The evaluation aims to provide the best estimate of the impact of the project on malnutrition outcomes in the community as well as outputs at the health centre level (malnutrition care outputs) and to describe quantitatively and qualitatively the changes that took place (or did not take place) within health centres as a result of the program.DiscussionAlthough PBF schemes are blooming in low in-come countries, there is still a need for evidence, especially on the impact of revising the list of remunerated indicators. It is expected that this impact evaluation will be helpful for the national policy dialogue in Burundi, but it will also provide key evidence for countries with an existing PBF scheme and confronted with malnutrition problems on the appropriateness to extend the strategy to nutrition services.Trial registrationClinicalTrials.gov PRS Identifier: NCT02721160; registered March 2016

A Large‐Scale Behavior Change Initiative to Improve Infant and Young Child Feeding Had Positive Impact on Language and Motor Development in Bangladesh

Promotion of adequate nutrition early in life through interventions aimed at improving infant and young child feeding (IYCF) has the potential to contribute to motor, cognitive, and social‐emotional development. This study evaluated this potential in Bangladesh where IYCF interventions were implemented in 300 sub‐districts through the Alive & Thrive (A&T) initiative.A cluster‐randomized, non‐blinded, impact evaluation design was employed comparing two intervention packages i.e., intensive interpersonal counseling on IYCF, mass media, and community mobilization vs. non‐intensive standard nutrition counseling and mass media. Twenty sub‐districts were randomized to either intensive or non‐intensive intervention areas. Cross‐sectional household surveys were conducted at baseline (2010) and four years later (2014) in the same communities. About 4,000 children aged 0–47.9 months in Bangladesh were surveyed at baseline and endline. Language development was measured with 21 items in which the mother was asked if her child had reached each milestone, ordered from least advanced to most advanced, resulting in a score from 0 to 21. Gross motor development was measured through mother report and observation with 29 ordered items, resulting in a score from 0 to 29. We derived difference‐in‐difference impact estimates using fixed‐effects regression models that assessed differences between the two intervention packages over time, adjusting for geographic clustering, infant age, gender, and covariates that differed between packages from baseline to endline.The difference‐in‐difference in language development scores between intensive and non‐intensive areas was 1.07 (p<0.001) among children 6–23.9 months and 0.78 (p<0.10) among children 24–47.9 months. For motor development, the difference‐in‐difference was 0.84 (p<0.10) among 6–23.9 month old children. From baseline to endline, the program had positive impact on language development among children 6–23.9 months (increase of 0.92 for intensive vs. decrease of 0.37 for non‐intensive) and a protective effect among children 24–47.9 months (increase of 0.07 for intensive vs. decrease of 0.87 for non‐intensive).The large‐scale intensive package in Bangladesh had substantial, significant impact on breastfeeding and complementary feeding practices (reported elsewhere). The intensive package, which promoted active, responsive feeding and strategies to manage poor appetite, had significant impact on language development and a marginally significant impact on motor development. Pathways to better development could include greater dietary diversity, more frequent feeding, and/or more effective strategies to manage poor appetite (all three could result in greater nutrient intake) and/or greater caregiver engagement with the child. Thus, both nutritional and behavioral mechanisms could be responsible for the improvements in child development; these mechanisms should be further studied.Support or Funding InformationFunding: Bill & Melinda Gates Foundation, through Alive & Thrive, managed by FHI 360; CGIAR Research Program on Agriculture for Nutrition and Health (A4NH), led by the International Food Policy Research Institute (IFPRI)

Impact of Six Years Community Directed Treatment with Ivermectin in the Control of Onchocerciasis, Western Ethiopia.

BackgroundThe African Program for Onchocerciais Control (APOC) with a main strategy of community directed treatment with ivermectin (CDTI) was established with the aim of eliminating Onchocerciasis as a disease of public health and socio-economic importance. The study area was a hyper endemic area just before the implementation of CDTI. It has been implemented for six years in this district but yet not been evaluated. So, the objective of this study was to evaluate the impact of six years CDTI on parasitological and clinical indices of OnchocerciasisMethodsThis study employed a pre-post impact evaluation design. The minimum sample size for this study was 1318; the respondents were selected by multi-stage sampling technique. Data on socio-demographic characteristics using a semi-structured questionnaire, clinical examination for skin signs and symptoms of Onchocerciasis and two bloodless skin snips from each side of the gluteal fold were taken from the entire study participants. SPSS version 16.0 and Medcalc version 12.2.1.0 were used for analysis.ResultThe microfilaridermia reduced from the pre-intervention value of 74.8% to 40.7%, indicating a 45.6% reduction, mean intensity from 32.1(SD = 61.5) mf/mg skin snip to 18.7(SD = 28.7)indicating 41.75% reduction, CMFL from 19.6 mf/mg skin snip to 4.7 indicating 76% reduction. The result also showed that microfilaridermia and mean intensity decreased as the number of treatment taken increased. Pruritis, leopard skin, onchocercomata and hanging groin reduced by 54.4%, 61.3%, 77.7% and 88.5% respectively.ConclusionsThe implementation of CDTI significantly reduced the parasitological and clinical indices of Onchocerciasis, so, efforts should be made to improve the annual treatment coverage and sustainability of CDTI to drastically reduce the micro filarial load to the level the disease would no longer be a public health problem.

Why so many “rigorous” evaluations fail to identify unintended consequences of development programs: How mixed methods can contribute

Many widely-used impact evaluation designs, including randomized control trials (RCTs) and quasi-experimental designs (QEDs), frequently fail to detect what are often quite serious unintended consequences of development programs. This seems surprising as experienced planners and evaluators are well aware that unintended consequences frequently occur. Most evaluation designs are intended to determine whether there is credible evidence (statistical, theory-based or narrative) that programs have achieved their intended objectives and the logic of many evaluation designs, even those that are considered the most “rigorous,” does not permit the identification of outcomes that were not specified in the program design. We take the example of RCTs as they are considered by many to be the most rigorous evaluation designs. We present a numbers of cases to illustrate how infusing RCTs with a mixed-methods approach (sometimes called an “RCT+” design) can strengthen the credibility of these designs and can also capture important unintended consequences. We provide a Mixed Methods Evaluation Framework that identifies 9 ways in which UCs can occur, and we apply this framework to two of the case studies.