Theory-driven Evaluation Research Articles

A process evaluation of a home garden intervention

BackgroundMost reviews of nutrition-sensitive programs assess the evidence base for nutrition outcomes without considering how programs were delivered. Process evaluations can fill this void by exploring how or why impacts were or were not achieved. This mid-term process evaluation examines a home garden intervention implemented in a large-scale, livelihoods improvement program in Odisha, India. The objectives are to understand whether the intervention was operating as planned (fidelity), investigate potential pathways to achieve greater impact, and provide insights to help design future home garden programs.MethodologyData collection and analysis for this theory-driven process evaluation are based on a program impact pathway that shows the flow of inputs, processes, outputs, outcomes, and impacts. Quantitative and qualitative data from focus group discussions, semi-structured interviews, and a Process Net-Mapping exercise with beneficiaries, frontline workers, and program management staff.ResultsDespite a mismatch between the design and implementation (low fidelity), the process evaluation identified positive outputs, outcomes, and impacts on home garden production, consumption, income, health and nutritional outcomes, and women’s empowerment. Flexibility led to greater positive outcomes on nutrition, the adoption of sustainable agricultural practices and easy-to-understand nutrition models, and the likelihood of the intervention being sustained after the program ends.ConclusionsTo help food systems in rural settings reduce food insecurity by utilizing more sustainable agricultural practices, we recommend that home garden interventions include instruction on easy-to-understand nutrition models and on how to make natural fertilizer. Finding local solutions like home gardens to help address critical supply issues and food insecurity is paramount.

Realist analysis of qualitative data in health professions education research.

Qualitative realist analysis is gaining in popularity in health professions education research (HPER) as part of theory-driven program evaluation. Although realist approaches such as syntheses and evaluations typically advocate mixed methods, qualitative data dominate currently. Various forms of qualitative analysis have been articulated in HPER, yet realist analysis has not. Although realist analysis is interpretive, it moves beyond description to explain generative causation employing retroductive theorising. Ultimately, it attempts to build and/or 'test' (confirm, refute or refine) theory about how, why, for whom, when and to what extent programs work using the context-mechanism-outcome configuration (CMOC) heuristic. This paper aims to help readers better critique, conduct and report qualitative realist analysis. We describe four fundamentals of qualitative realist analysis: (1) simultaneous data collection/analysis; (2) retroductive theorising; (3) configurational analysis (involving iterative phases of identifying CMOCs, synthesising CMOCs into demi-regularities and translating demi-regularities into program theory); and (4) realist analysis quality (relevance, rigour, richness). Next, we provide a critical analysis of realist analyses employed in 15 HPER outputs-three evaluations and 12 syntheses. Finally, drawing on our understandings of realist literature and our experiences of conducting qualitative realist analysis (both evaluations and syntheses), we articulate three common analysis challenges (coding, consolidation and mapping) and strategies to mitigate these challenges (teamwork, reflexivity and consultation, use of data analysis software and graphical representations of program theory). Based on our critical analysis of the literature and realist analysis experiences, we encourage researchers, peer reviewers and readers to better understand qualitative realist analysis fundamentals. Realist analysts should draw on relevant realist reporting standards and literature on realist analysis to improve the quality and reporting of realist analysis. Through better understanding the common challenges and mitigation strategies for realist analysis, we can collectively improve the quality of realist analysis in HPER.

The Garden of Evaluation Approaches Visualization

Background: The Garden of Evaluation Approaches (Montrosse-Moorhead, et al., 2024a) maps evaluation approaches against eight dimensions of practice and situates them in their philosophical orientations and methodological dispositions. Purpose: The Garden’s guiding question is: How do evaluation approaches compare in terms of dimensions that facilitate use and application? Primary intended users are evaluation practitioners, and secondary intended users include ecosystem actors (e.g., clients, program participants, funders), educators, researchers, and theoreticians. Setting: Not applicable. Intervention: Not applicable. Research Design: Exploratory sequential mixed methods. Data Collection and Analysis: Data collection and analysis included locating and reading primary sources for each evaluation approach. One team member thematically coded each source for evidence of an approach’s stance on the eight dimensions of practice, its’ philosophical orientation and methodological disposition, and steps for implementing the approach in practice. This qualitative evidence was used to generate ordinal ratings for the eight dimensions using a rating scale developed by the team, and to classify the approaches' philosophical orientation and methodological disposition. Drafts of handouts summarizing qualitative and quantitative data were reviewed by the two other team members, with dialogue and deliberation among all members used to come to consensus. Findings: The first iteration of the Garden includes seven approaches: fourth generation evaluation, made in Africa, nation-to-nation evaluation, practical participatory evaluation, sistematización de experiencias, theory-driven evaluation, and transformative participatory evaluation. Each approach is represented as a flower within the Garden, with each petal corresponding to the approaches’ stance on one of the eight dimensions of practice. The colors of the flowers represent the underlying philosophical foundation of the approach. The pattern within the central disc of each flower represents the methodological disposition encouraged by the approach. Summaries of the empirical mixed-method analysis are made transparent in supporting handouts. Future work will grow flowers for additional approaches and continue to map out the Garden.

Imprinting and the evolution of evaluation: A descriptive account of social impact evaluation methodological practice

This study investigates the evolution of evaluation approaches within the charitable sector, emphasising the proliferation of options throughout waves of evaluation diffusion. We draw on imprinting theory to describe the persistence of specific practices despite the emergence of alternatives. The research aims to examine the imprinted values and practices of earlier evaluation waves on the current practice of evaluation by documenting the current methodological practice in a large sample of evaluation reports. The study illuminates the enduring impact of theory-driven evaluation, introduced in the 1980s, on current practices. However, it also reveals how this approach has evolved over subsequent evaluation waves, acquiring divergent characteristics. This imprinting is not only due to effectiveness but also due to normative pressures, which prioritise conformity with norms over practical utility. The study highlights the potential consequences of this imprinting, including management-focused evaluations and mission drift. We conclude by offering practical options to address tensions within the field and call for further research to better understand the nuances of decision-making in evaluation approaches.

Heterogeneous pathways of technological change in marginalised rural areas: the case for fuller accounts of adoption

ABSTRACT Dynamic and cumulative processes shaping technology adoption receive less attention in agricultural research compared to snapshot analyses of its determinants. We address this through theory-driven evaluation and the Propositions, Encounters, Dispositions and Responses framework. In 2005, research institutes introduced improved cassava varieties (ICVs) in East Cameroon. In the following decade, economic transformations brought in new development actors who revived their circulation. The interactions and actors who intervened in a relatively short period and the diversity of adoption pathways in a relatively small area highlight the complexity of technological change. Learning-oriented, systemic and systematic evaluations are needed for fuller adoption accounts.

Evaluating a vertical nurse-led service in the integration of palliative care in a tertiary academic hospital.

Groote Schuur Hospital is a large Academic Hospital in South Africa that is in the process of integrating palliative care (PC) via a vertical nurse-led doctor-supported (VNLDS) service that was initially established to deliver clinical care. PC integration should occur across multiple dimensions and may result in variable degrees of integration between levels of the healthcare system. This research evaluates the VNLDS through a theory-driven evaluation to describe how the service affected integration. A mixed-method sequential design consisting of a narrative literature review on the theory of integration and PC, retrospective quantitative data from a PC service delivery database, qualitative data from semi-structured interviews and document analyses. It was structured in three phases which assisted in confirming and expanding the data. Statistical analyses, deductive thematic coding and documentary analyses were conducted according to the conceptual framework of PC integration. The PC integration process was facilitated in the following ways: (i) the service provided good clinical PC; (ii) it was able to integrate on a professional level into specific diseases, such as cancer but not in all diseases; (iii) developing organizational structures within the service and (iv) the observed benefit of good clinical care increased the value stakeholders assigned to PC, thereby driving the adoption of PC. However, there are still clinicians who do not refer to PC services. This gap in referral may be grounded in assumptions and misconceptions about PC, especially at the organizational level. Observed PC service delivery is core to integrating PC across the healthcare system because it challenges normative barriers. However, the VNLDS could not achieve integration in leadership and governance, education and hospital-wide guidelines and policies. Whole system integration, foregrounding organizational commitment to PC excellence, is core to integrating PC. The VNLDS service has effectively linked PC in specific disease profiles and normalized the PC approach where healthcare workers observed the service. These integrational gaps may be grounded in assumptions and misconceptions about PC, especially at the organizational level.

Implementing the QUALI-DEC project in Argentina, Burkina Faso, Thailand and Viet Nam: a process delineation and theory-driven process evaluation protocol

ABSTRACT The project ‘Quality Decision-making by women and providers’ (QUALI-DEC) combines four non-clinical interventions to promote informed decision-making surrounding mode of birth, improve women’s birth experiences, and reduce caesarean sections among low-risk women. QUALI-DEC is currently being implemented in 32 healthcare facilities across Argentina, Burkina Faso, Thailand, and Viet Nam. In this paper, we detail implementation processes and the planned process evaluation, which aims to assess how and for whom QUALI-DEC worked, the mechanisms of change and their interactions with context and setting; adaptations to intervention and implementation strategies, feasibility of scaling-up, and cost-effectiveness of the intervention. We developed a project theory of change illustrating how QUALI-DEC might lead to impact. The theory of change, together with on the ground observations of implementation processes, guided the process evaluation strategy including what research questions and perspectives to prioritise. Main data sources will include: 1) regular monitoring visits in healthcare facilities, 2) quantitative process and output indicators, 3) a before and after cross-sectional survey among post-partum women, 4) qualitative interviews with all opinion leaders, and 5) qualitative interviews with postpartum women and health workers in two healthcare facilities per country, as part of a case study approach. We foresee that the QUALI-DEC process evaluation will generate valuable information that will improve interpretation of the effectiveness evaluation. At the policy level, we anticipate that important lessons and methodological insights will be drawn, with application to other settings and stakeholders looking to implement complex interventions aiming to improve maternal and newborn health and wellbeing. Trial registration: ISRCTN67214403.

Four suppositions about emergent properties of complex interventions that operate and function as systems: A call for evaluation research

This article focuses on evaluating the emergent properties of complex interventions that operate and function as systems. After defining key terms, four suppositions of emergence are presented. The suppositions are based on the authors’ reflections on evaluating several complex interventions using a systems approach. Supposition 1 posits that there is an operational emergent property, paralleling the theory-driven evaluation concept of implementation theory noted by Carol Weiss. Supposition 2 suggests that every complex intervention also has a functional emergent property that speaks to the effectiveness of the intervention. Several important distinctions between a functional emergent property and the long-term outcomes derived based on a reductionist worldview are noted. Supposition 3 builds on the work of Chalmers to posit that it is possible for a complex intervention to simultaneously have more than one operational emergent property and more than one functional emergent property. Supposition 4 is grounded in the system definition to propose that operational emergence is a prerequisite for functional emergence. Suppositions by definition require proof to become theories. The article calls for evaluators to share whether their experiences support these suppositions and encourages investments in evaluation research to test them.

Courage, Honesty, and Evaluation in the Apprehensive University

A consistent question in education is how to evaluate the degree to which universities and their programs are meeting the claims they make on their webpages and other materials, which entice students and faculty alike to join their collegiate community. Misalignments between what is promised and what is provided harm all community members but have disproportionate effects on students of color. It is therefore an ethical imperative for the higher education sector to undertake system wide evaluations because of the ever-rising financial and emotional costs of graduate education. For educators and administrators alike, this means systematically interrogating data to identify unseen patterns, challenge assumptions, and ask both critical and highly uncomfortable questions; for educators, this may include a truthful assessment of our own practices and assumptions. We propose drawing from the field of program evaluation and using theory-driven evaluation as a specific framework to understand graduate education process and outcomes. This conceptual paper links together existing literatures and is augmented by the authors’ reflection and dialogue about their experiences designing and implementing graduate education across several institutions. We end with a call for courage and honesty in carefully evaluating graduate education for the betterment of all students, faculty, and administrators.

Using contribution analysis to evaluate health professions and health sciences programs.

Course evaluation in health education is a common practice yet few comprehensive evaluations of health education exist that measure the impact and outcomes these programs have on developing health graduate capabilities. To explore how curricula contribute to health graduate capabilities and what factors contribute to the development of these capabilities. Using contribution analysis evaluation, a six-step iterative process, key stakeholders in the six selected courses were engaged in an iterative theory-driven evaluation. The researchers collectively developed a postulated theory-of-change. Then evidence from existing relevant documents were extracted using documentary analysis. Collated findings were presented to academic staff, industry representatives and graduates, where additional data was sought through focus group discussions - one for each discipline. The focus group data were used to validate the theory-of-change. Data analysis was conducted iteratively, refining the theory of change from one course to the next. The complexity in teaching and learning, contributed by human, organizational and curriculum factors was highlighted. Advances in knowledge, skills, attitudes and graduate capabilities are non-linear and integrated into curriculum. Work integrated learning significantly contributes to knowledge consolidation and forming professional identities for health professional courses. Workplace culture and educators' passion impact on the quality of teaching and learning yet are rarely considered as evidence of impact. Capturing the episodic and contextual learning moments is important to describe success and for reflection for improvement. Evidence of impact of elements of courses on future graduate capabilities was limited with the focus of evaluation data on satisfaction. Contribution analysis has been a useful evaluation method to explore the complexity of the factors in learning and teaching that influence graduate capabilities in health-related courses.

Exploring patterns of collaboration among professionals, organisations, patients and communities that meet complex challenges and promote patient well-being

Managing the well-being of a super-ageing society requires effective collaboration and communication between professionals, organisations, patients and families. Alongside education and training, the importance of a comprehensive understanding of interprofessional competencies cannot be understated. Professor Junji Haruta is building on his personal experience as a physician dealing with family illnesses to conduct research that promotes a holistic approach to medicine. He believes primary care professionals will play a crucial role in providing optimal care for the elderly population in communities for the next 20 years. Haruta is developing resources, including models and theories linked to education on complex issues in primary care, to promote effective interprofessional education. These can be utilised by individuals, universities, organisations and the wider community as a comprehensive system and facilitate the development of solutions to the increasingly complex challenges associated with healthcare, promoting individual and societal well-being. Haruta employs empirical generalisation, which explores relationships between two or more variables observed across different conditions and involves collecting findings from various cases and specific contexts to identify patterns. In one study, he and his collaborators investigated the learning process of medical students in an online community diagnostics programme using a realist approach, which involves theory-driven evaluation, to identify patterns of learning within this programme using the Context (C), Mechanism (M) and Outcome (O) framework. They found that the students acquired knowledge about communities through four distinct learning patterns, and the community diagnosis programme facilitated a deeper understanding of communities through the application of discovery learning, variation theory and cognitive flexibility theory.

The use of the Capability-Opportunity- Motivation Behavior (COM-B) model to identify barriers to medication adherence and the application of mobile health technology in adults with coronary heart disease: A qualitative study

ObjectiveAmong patients with coronary heart disease, we sought to address the research questions of: 1) What is the acceptability of applying a technology-enabled approach to support medication adherence?; and 2) What are barriers to medication adherence using the Capability-Opportunity-Motivation Behavior (COM-B) model as a guiding framework? MethodsApplying qualitative research methods, we employed a series of 3 focus groups per individual (total 9 sessions). Coded data from thematic analysis were mapped to the COM-B model components for meaningful associations. ResultsFourteen participants were recruited (median age 69.5 ± 11, 50% female). Barriers to medication adherence were organized along these COM-B domains: psychological capability (forgetfulness, distractions, fear of side effects), physical opportunity (inaccessible medications, inability to renew prescriptions), reflective (burdening family members), and automatic motivation (medication fatigue, health decline). ConclusionsTailored text messaging and mobile phone apps were perceived as helpful tools for medication adherence. The COM-B model was useful to provide a comprehensive, theory-driven evaluation of patients' beliefs and motivations on whether to engage in medication adherence. InnovationTo date, text messaging and mobile applications have not been widely implemented in the clinical setting and provide a major opportunity to innovate on approaches to address medication adherence.

Group-based circuit training to improve mobility after stroke: a cross-sectional survey of German and Austrian physical therapists in outpatient settings.

A contextual transferability analysis identified group-based circuit training (GCT) as an optimal intervention in German and Austrian outpatient physical therapy to improve mobility post-stroke. GCT incorporates task-oriented, high-repetitive, balance, aerobic and strength training and allows for increased therapy time without increasing personnel. To determine the extent to which German and Austrian physical therapists (PTs) use GCT and its components in the outpatient treatment of stroke-related mobility deficits and to identify factors associated with using GCT components. A cross-sectional online survey was conducted. Data were analyzed descriptively and using ordinal regression. Ninety-three PTs participated. None reported using GCT moderately to frequently (4- 10/10 patients). The percentage of PTs reporting frequent use (7- 10/10 patients) of task-oriented, balance, strength, aerobic, and high-repetitive training was 45.2%, 43.0%, 26.9%, 19.4%, and 8.6%, respectively. Teaching or supervising students, time for evidence-based practice activities at work, and working in Austria was associated with using GCT components frequently. German and Austrian PTs do not yet use GCT in outpatient physical therapy for stroke. Almost half of PTs, however, employ task-oriented training as recommended across guidelines. A detailed, theory-driven and country-specific evaluation of barriers to GCT uptake is necessary to inform implementation.

Impact of a Regional Grant Program Through the Lens of Social Cognitive Career Theory: A Mixed-Method Evaluation.

Evaluations of educational grant programs have focused on research productivity, with few examining impacts on grantees or effective program characteristics. This evaluation examined the regional grant program sponsored by Group on Educational Affairs to examine if and how grantees' careers were affected by funding, and if these experiences aligned with program goals. In this concurrent, mixed-methods theory-driven evaluation, quantitative and qualitative data were analyzed independently and then integrated to examine complementarity. Quantitative data examined differences among 4 geographic regions and included proposal and grantee characteristics abstracted from administrative records of 52 funded proposals from 2010-2015 grant cycles. Qualitative data from 23 interviews conducted from 2018 to 2019 explored the impact on grantees, with Social Cognitive Career Theory (SCCT) serving as a framework for deductive thematic analysis. To facilitate integration of findings, quantitative data were layered onto each interview to permit exploration of associations between the 2 data types. Although significant regional differences existed in project length and amount of funding, there were few regional differences in grantee experiences. Despite small funding amounts, grants were perceived as career launching pads. The SCCT framework accounted for grantee experiences, including researcher identity formation and subsequent research, but did not capture collaboration phenomena. Integration of the 2 data types identified experience patterns unique to different groups of grantees (e.g., more or less research experience). The diversity among grantees suggests that clarification of program goals and stronger alignment with criteria for funding may be warranted. This evaluation illuminates why small educational grant programs may or may not impact interest and productivity in research. Implications exist for funders, including clarifying program goals and providing support for less experienced grantees. Future research should explore grantee subsets (e.g., underrepresented in medicine) to further identify what fosters or inhibits careers of medical education scholars.

Prospects for applying a theory of change model to the use of research evidence in autism education

Educators and educational researchers show continued interest in how schools can best make use of research evidence in bringing about change in practice in schools. A number of models have been developed to support schools in this challenge, such as research learning communities and lesson study. However, questions remain about the effectiveness of such models, their fit to the particular needs of schools and the extent to which they contribute meaningfully to the body of evidence used to inform changes to practice within the field of education. This issue is of particular relevance when considering the inclusion of autistic children in the classroom partly because of the large body of research being undertaken on autism across a range of domains with varying epistemological perspectives (e.g., neuroscience, psychology, pedagogy) and partly due to the widespread need to support autistic children in the classroom. Questions have also been raised about the evidence policy “agenda,” particularly in terms of reliance on positivist models centered on randomized controlled trials. These concerns focus on the extent to which performative or neoliberal perspectives on effectiveness might mask the complexity of how practice and knowledge (or evidence) are related in models of teacher professional working. One particular approach that could have potential in addressing these is that of Theory of Change (ToC). ToC models come from the field of theory-driven evaluation and draw on frameworks for relating practice to knowledge such as realist evaluations whereby the evaluation focuses on understanding how complex programs work in specific contexts by examining the mechanisms that lead to particular outcomes. ToC models consider under what conditions, for whom, and for what reasons or aims a given activity will achieve its intended outcomes. This paper considers the scope for the application of ToC models by reviewing a selected case from a completed study on the implementation of models for developing evidence informed practice in schools for autism education. By applying a ToC lens to what did happen in this case, we will “re-imagine” this case from a ToC perspective. This approach will serve to illustrate the possibilities for how ToC models could be used in future practice to advance evidence-informed practice in autism education.

Defining a Treatment Model for Self-Management of Fatigue in Rehabilitation of Acquired Brain Injury Using the Rehabilitation Treatment Specification System.

Systematic treatment descriptions to standardize and evaluate management of fatigue after acquired brain injury (ABI) are lacking. The purpose of this multi-phase qualitative study was to formulate a treatment model for promoting self-management of fatigue in rehabilitation of ABI based on practice-based understandings and routines. The study was conducted in a community-based rehabilitation center in Denmark. The model was defined using the Rehabilitation Treatment Specification System. Phase 1 comprised co-production workshops with five service providers (occupational therapists, physiotherapists, and a neuropsychologist) to elicit preliminary treatment theories. In Phase 2, four case studies were conducted on management of fatigue in vocational rehabilitation. Interviews (n = 8) and treatment log entries (n = 76) were analyzed thematically to specify treatment targets and active ingredients. The treatment model comprised five main components: (i) Knowledge and understanding of fatigue, (ii) Interoceptive attention of fatigue, (iii) Acceptance of fatigue, (iv) Activity management, and (v) Self-management of fatigue. For each component, lists of targets and active ingredients are outlined. In conclusion, management of fatigue includes multiple treatment components addressing skills, habits, and mental representations such as knowledge and attitudes. The model articulates treatment theories, which may guide clinical reasoning and facilitate future theory-driven evaluation research.

Ensuring that a school-based smoking cessation program for adolescents is successful: A realist evaluation of the TABADO program and the program theory.

A smoking cessation program for adolescents, TABADO, demonstrated its effectiveness following a controlled trial conducted in 2007/2009. The program is now being scaled up nationally. In order to retain its efficacy across the diversity of contexts in the generalization process, we needed to assess the processes and mechanisms that inform its effects. Theory-driven evaluation is one approach used to address these issues. The aim of the present research is to develop the TABADO program theory. More specifically, we attempt to identify the factors and mechanisms that promote or hinder the enrollment and retention of student smokers in the program. We conducted a realist evaluation of the TABADO program through 1) a documentary analysis to construct the initial program theory, and 2) a multiple case study (n = 10) conducted in three regions in France to test and enrich the initial theory with contextual, organizational and mechanistic components. We used the Intervention-Context-Actors-Mechanisms-Outcomes configurations to guide our analysis and to present our results. Our analysis highlighted 13 mechanisms that foster the enrollment and retention of student smokers in the TABADO program (e.g., being prepared to quit smoking, feeling encouraged in the attempt to quit smoking). To activate these mechanisms, the involvement of various actors is required (e.g., the school nurse, teachers), together with a combination of interventional and contextual factors (e.g., confidentiality, informal speaking time). These findings allowed us to transform the TABADO program into a new optimized strategy, TABADO2, which is theory-based. Our research helps to explain why adolescent smokers enroll and stay in a school-based smoking cessation program. TABADO2 needs to be considered in a more comprehensive way than the original research-based TABADO, and should be tailored to its implementation context.

Examining the Impact of Teacher Learning Communities on Self-Efficacy and Professional Learning: An Application of the Theory-Driven Evaluation

Collaborative professional learning is essential for teachers to work collectively in facilitating student learning. A program of Learning Community under Leadership for Learning was launched to support teacher learning in authentic situations in Taiwan. Applying the theory-driven evaluation model, the study aimed to investigate the impact of a program component of teacher learning communities (TLCs) on professional learning beliefs and behaviors. A sample of 226 elementary and junior high school teachers in Taiwan was surveyed and analyzed using structural equation modeling. The findings suggest that the experiences in TLCs had an impact on teachers’ professional learning beliefs and behaviors through their self-efficacy. Additionally, teacher self-efficacy was found to have a direct impact on their beliefs and behaviors, and beliefs were found to affect behaviors significantly. This study has expanded the understanding of self-efficacy in relation to professional learning beliefs and behaviors and provided practical insights for effective strategies for teacher development.

Causality and Complexity in Evaluating Equity Interventions: Conceptual Issues That Need to Be Addressed in Theory-Driven Evaluation Approaches

We discuss how explicit thinking about a variety of causal pathways, informed by a dynamic systems lens that is responsible for exacerbating and diminishing inequities as well as different types of complexities (related to program pathways), will further help develop theory-driven evaluation approaches such as Contribution Analysis. We argue that contribution claims associated with interventions focused on addressing inequities need to consider the multiple types of causal pathways by which a program can help reduce inequities.

Simple but not simplistic: Findings from a theory-driven retrospective evaluation of a small projects program

Background and purposeFrom 2010–2019, the United States Peace Corps Volunteers in Georgia implemented 270 small projects as part of the US Peace Corps/Georgia Small Projects Assistance (SPA) Program. In early 2020, the US Peace Corps/Georgia office commissioned a retrospective evaluation of these projects. The key evaluation questions were: 1) To what degree were SPA Program projects successful in achieving the SPA Program objectives over the ten years, 2) To what extent can the achieved outcomes be attributed to the SPA Program’s interventions, and 3) How can the SPA Program be improved to increase likelihood of success of future projects. MethodsThree theory-driven methods were used to answer the evaluation questions. First, a performance rubric was collaboratively developed with SPA Program staff to clearly identify which small projects had achieved intended outcomes and satisfied the SPA Program’s criteria for successful projects. Second, qualitative comparative analysis was used to understand the conidtions that led to successful and unsuccessful projects and obtain a causal package of conditions that was conducive to a successful outcome. Third, causal process tracing was used to unpack how and why the conjunction of conditions identified through qualitative comparative analysis were sufficient for a successful outcome. FindingsBased on the performance rubric, thirty-one percent (82) of small projects were categorized as successful. Using Boolean minimization of a truth table based on cross case analysis of successful projects, a causal package of five conditions was sufficient to produce the likelihood of a successful outcome. Of the five conditions in the causal package, the productive relationship of two conditions was sequential whereas for the remaining three conditions it was simultaneous. Distinctive characteristics explained the remaining successful projects that had only several of the five conditions present from the causal package. A causal package, comprised of the conjunction of two conditions, was sufficient to produce the likelihood of an unsuccessful project. ConclusionsDespite having modest grant amounts, short implementation periods, and a relatively straightforward intervention logic, success in the SPA Program was uncommon over the ten years because a complex combination of conditions was necessary to achieve success. In contrast, project failure was more frequent and uncomplicated. However, by focusing on the causal package of five conditions during project design and implementation, the success of small projects can be increased.