Evaluation Capacity Building Research Articles

Leveraging the Kirkpatrick four‐level model to evaluate evaluation capacity building work

AbstractIn this article, we reflect on a decade of using the Kirkpatrick four‐level model to evaluate a multifaceted evaluation capacity building (ECB) initiative. Traditionally used to assess business training efforts, the Kirkpatrick model encourages evidence to be gathered at four levels: reaction, learning, behavior, and results. We adapted these levels to fit the context and information needs of the EvaluATE project, an ECB initiative funded by the National Science Foundation. As members of the external evaluation and project teams, throughout the article we describe how each level was modified and translated into evaluation questions. Our adapted Kirkpatrick levels are implementation and reach, satisfaction, learning, application, and impact. Using these adapted Kirkpatrick levels to ground our evaluation challenged us to integrate multiple data sources to tell a comprehensive story that served the information needs of the project team and the funder. Overall, we found the Kirkpatrick model to be practical, accessible, and flexible, allowing us to capture the multidimensional aspects of the ECB initiative. However, there are opportunities to enhance the utility of the Kirkpatrick framework by integrating other evaluation approaches, such as culturally responsive and equitable evaluation and principles‐focused evaluation.

Evaluation capacity building in theory and practice: Revisiting models from practitioner perspectives

AbstractThis article seeks to bring practitioner experience to bear on existing models of ECB with a particular focus on the models through the lens of our own ECB practice. We reflect on how our ECB practices align with or challenge these models, and how the insights that come from those reflections can inform future ECB research and frameworks for evaluating ECB initiatives. As is often the case when theory collides with practice, current models may not always reflect and serve the work at hand, and the value and usefulness, as well as accuracy and relevance, of existing models is worth investigating. With this in mind, we offer input to inform future models of ECB that are more inclusive of and relevant for the broad spectrum of current ECB practice and, subsequently, its evaluation.

Epilogue: Looking forward

AbstractInspired by recent advancements in evaluation capacity building (ECB), we developed an issue of New Directions for Evaluation to respond to the call to action made by fellow researchers and practitioners to leverage evaluation to expand upon our understanding of meaningful ECB and share practical lessons learned. The resulting issue brought practitioners and scholars together to reflect on and share their contributions to knowledge around the evaluation of ECB as they pertain to research, theory, methodological approaches, and practical applications. This epilog highlights key themes and practical lessons that can support others in their approaches to ECB evaluation. As this issue aims to continue, and in some cases kick‐start, conversations about the evaluation of ECB, future directions for research and practice are explored. We conclude with a call to action, urging those engaging in ECB to plan to evaluate their important initiatives and share relevant findings to others in the field.

Using contribution analysis to assess evaluation capacity outcomes in community organizations

AbstractCommunity organizations face challenges in addressing the needs of their members and beneficiaries, particularly during times of crisis. More than ever, evidence‐informed programming and decision‐making are essential to help organizations “do more with less”; however, community organizations seldom have the individual and organizational evaluation capacity required to conduct and use evaluations to inform decision‐making. This article provides an overview of the evaluation of the LaboEval evaluation capacity building (ECB) intervention, a 5‐year program implemented in 16 organizations. LaboEval brings together direct and indirect individual and organizational ECB activities to enhance evaluation capacity. Contribution analysis (CA) was used as the evaluation approach in this study, assessing the intervention's activities, outputs, outcomes, and assumptions. Data sources include interviews, document reviews, surveys, administrative data, and a literature review. This evaluation contributes to the field by applying CA to a multiyear, multi‐organizational ECB intervention. The findings inform evaluation capacity builders and evaluators interested in CA, showcasing a unique approach to evaluating complex interventions across multiple organizations.

Communities of practice as an effective ECB approach for nonevaluation specialists: A case study in an Australian state government department

AbstractAmong the various evaluation capacity building strategies, there is an important opportunity to enhance the evidence on the effectiveness of communities of practice (CoPs). We argue that evaluation CoPs have great potential to develop evaluative thinking among nonevaluation specialists and build the foundations of a sustainable organizational evaluation culture. This article presents a case study of an evaluation community of practice in an Australian state government agency, with evidence collected over 4 years against a comprehensive evaluation framework. It outlines in detail an empirically tested evaluation‐CoP good practice from the initial design and logic model to implementation and measurement against individual, group, and organizational outcomes.

Two decades of evaluating evaluation capacity building: A bibliographic coupling review

AbstractEvaluation capacity building (ECB) has emerged as a dominant tool in evaluation. Several reviews have taken stock of ECB over the past 20 years, examining how understanding of ECB has developed over time. By examining this existing research on the evaluation of ECB design, effectiveness, measurement, integration, and sustainability, opportunities to advance the field's existing knowledge base around ECB evaluation forward become apparent. This landscape review uses bibliographic coupling and content analysis to offer a robust analysis of publications on the evaluation of ECB. The results of this review of multidisciplinary ECB publications shed light on areas of strength within the existing literature and identify opportunities to build upon current scholarship. Promising areas for future research include exploring how moderators and mediators influence ECB outcomes, eliminating threats to internal validity in ECB studies, and strengthening the rigor required for external generalizability.

Blending emergent design evaluation with developmental evaluation principles in evaluation capacity building: Reflections on the Data for Social Impact initiative

AbstractIn addition to the dearth of literature on successful evaluation capacity building (ECB), there is also a need to highlight novel and blended approaches to evaluating ECB. To evaluate the Data for Social Impact (DSI) initiative, we used developmental evaluation and emergent design evaluation. The DSI initiative was created in the St. Louis metropolitan region in Missouri to build data capacity and collaboration across sectors to improve impact. Within its overarching goal of social‐sector data capacity building, DSI aims to amplify ECB both within and across organizations. In this article, we discuss the background of the initiative and our process of evaluation. This article offers an approach to evaluating ECB that allows for the flexibility and complexity that often comes with evaluation done in collaborative, real‐world contexts. We champion using an emergent and developmental evaluation principles to evaluate ECB programs to extend their reach beyond the evaluation experts and into organizations and communities working toward equity and justice.

Breaking the Mould? Expanding our Conceptualization of Evaluator Competencies

Evaluation capacity building (ECB) is an area of practice that many evaluators engage in. Previous ECB scholars have described ECB as distinct from the practice of evaluation, although the two are related. This research empirically examined competencies needed for practicing ECB, why those competencies are important, and how expert practitioners obtained those competencies. The findings suggest the role of an evaluation capacity builder is different from the role of an evaluator. The authors attempt to make the case that recognizing this difference is critical for the professionalization of the field.

E-Eval: Learnings from a Community-Based Evaluation Capacity Building Initiative

Evaluation capacity building (ECB) initiatives are important for supporting social-serving organizations in using evaluation to demonstrate and improve the effectiveness of their programs. In this practice note, the authors present their experience developing and implementing E-Eval, a responsive community-based course for building evaluation capacity in Alberta’s mentoring sector. In order to support others in taking a similar approach to building evaluation capacity in the social-serving sector, they describe how the concept of E-Eval emerged, outline its design and implementation processes, and discuss their learnings and their relevance for other professionals implementing or researching community-based ECB.

Developing an Evaluation Training Program for Community-Based Organizations: A Participatory Curriculum Development Approach

Community-based organizations increasingly recognize the value of evaluative insights in informing decision-making and improving their services. However, these organizations often lack the capacity to conduct and use evaluations, and traditional evaluation capacity building (ECB) activities are not always adapted to their particular contexts. This practice note introduces an innovative evaluation training program for small grassroots community-based organizations. Employing a participatory curriculum development approach, the training program was tailored to meet the specific needs and realities of the community sector. The authors first outline the collaborative process used to involve community and academic stakeholders in the program’s development. They then present the resulting training plan, consisting of seven self-directed online learning modules with chosen activities to foster learning and knowledge transfer into practice. The potential implications for designing, researching, and evaluating similar initiatives are explored and discussed.

Engaging evaluation champions: strategies to build evaluation capacity and promote data-informed decision making in youth development

Youth development researchers and practitioners share an interest in ensuring that youth development programs contribute positively to youth outcomes. Engaging in evaluation and data-informed decision making (DIDM) has the potential to empower practitioners to improve and adapt programs, improving youth outcomes. Yet, not all practitioners are comfortable engaging in evaluation and/or DIDM. Engaging in partnerships and utilizing internal supporters of evaluations (or “champions”) have been identified as potential strategies to build evaluation capacity and strengthen DIDM within an organization. However, little research explores how to do so. This study engaged evaluation champions in four states to examine their experience as they partnered with practitioners within their organization. Results suggest that peer groups can be utilized to promote evaluation capacity, especially by utilizing peers that already have an interest in evaluation. Practitioner engagement can also be developed by using less academic jargon in communication, highlighting the practical value of evaluation, and building capacity slowly.

The Science and Practice of Evaluation Capacity Building

Evaluation capacity building (ECB) has become an integral part of the practice and profession of evaluation. Over 20 years, much has been learned about ECB, yet substantial room exists for continuing to expand our ECB scholarship and practice. In this special issue, the authors share insights from recent scholarship and training efforts that can serve as a launching point for the next generation of ECB.

Concluding Remarks: A Bright Future for Evaluation Capacity Building

Evaluation capacity building (ECB) seeks to provide individuals and organizations with the knowledge, skills, and systems that they need to make evaluation an ongoing part of their everyday activities. This special issue sought to advance ECB by bringing to light original empirical research as well as practical applications of various ECB approaches. This concluding article provides an overview of the eight contributions comprising the special issue, highlighting lessons learned and future directions for ECB research and practice.

Strengthening Evaluation Capacity Building Practice Through Competition: The Max Bell School of Public Policy’s Evaluation Capacity Case Challenge

Despite the attention evaluation capacity building (ECB) has received over the past several decades, surprisingly few opportunities for learning about ECB exist. In response to this need, the Max Bell School of Public Policy at McGill University, in collaboration with ECB scholars across Canada, created a case competition focused exclusively on ECB—the Evaluation Capacity Case Challenge (EC3). Twenty individuals interested in learning about ECB and one organization (case site) interested in enhancing their existing evaluation capacity were selected to participate through a competitive application process. Participants attended a series of online workshops and engaged with an ECB coach to hone their skills and then took part in a two-day hybrid case challenge event where they had 24 hours to craft an ECB plan in response to a specific case challenge question presented by case site representatives. In this article, the authors describe EC3 in detail and share some key reflections from the inaugural year.

Exploring the Edges: Identifying the Next Generation of Evaluation Capacity Building Research and Practice Through Adjacency

Evaluation capacity building (ECB) is a well-established domain of scholarship and practice within the broader field of evaluation. Stemming from a concern to ensure sustainable and useful evaluation practices in organizations, ECB draws on several evaluation theories and approaches such as collaborative evaluation and utilization-focused evaluation. In a recent integrative review, the authors and their colleagues identified other theories, frameworks, and models that inform ECB; many of these were developed in other fields, such as social sciences and public administration. The present exploratory study sought to identify some of the knowledge and theories that inform ECB from other fields through a review of 47 “ECB-adjacent” articles. This article describes the various themes and topics included in these articles to formulate potential future avenues for ECB research and practice.

Meeting the Challenge: How the City of Kingston Is Working to Propel Evaluation Growth

This practice note shares key learnings from the inaugural Evaluation Capacity Case Challenge (EC3) held by the Max Bell School of Public Policy in April 2023. The purpose is to give readers an opportunity to consider how to shape evaluation capacity in general and within a municipal context. Evaluation capacity building (ECB) is a multifaceted concept to support shared learning about and understanding of evaluation. Strands of learning about ECB emerged from EC3 that can be applied within organizations. These centre on three important actions: (1) establishing a community of practice led by champions, (2) scaling and sequencing evaluation capacity, and (3) developing data literacy as a technical skill. The ECB strategies developed in EC3 offered new ways for the City of Kingston to consider and reconsider how it conducts, shares, learns from, and uses evaluation. Here, the authors describe evaluation within the City of Kingston, the city’s goals in entering EC3, key learnings from the challenge, and the city’s ensuing efforts to propel the growth of evaluation.

Evaluation Capacity Building: Experiential Learning Through Community–University Collaboratives

There is a growing need for evaluation capacity building (ECB) in community organizations; however, learning opportunities for both organizations and post-secondary students are limited, particularly opportunities to apply evaluation concepts in practice. This practice note describes four initiatives and highlights the ways in which experiential learning can be leveraged through community-university collaborations to provide responsive, hands-on ECB for both students and community organizations. The authors discuss the unique contributions of these initiatives and considerations for designing and implementing similar initiatives.

Evaluation Literacy and Use in Community Organizations: Insights and Implications for Evaluation Capacity Building

Past research on evaluation capacity in community organizations has focused on their capacity to conduct rather than use evaluation. This study sought to address this gap by exploring evaluation use and the supporting processes, systems, and structures in the community sector. The authors administered an online survey to 102 individuals working or involved in community organizations to measure their perceptions of their organization’s capacity to use evaluation. The survey was based on an adaptation of Bourgeois and Cousins’ evaluation capacity framework and a related assessment tool, the Organizational Evaluation Capacity Assessment Instrument, for the community sector context. Their findings offer empirical insights into three critical dimensions of the adapted framework associated with evaluation use in community organizations: evaluation literacy, organizational decision-making, and learning benefits. This exploratory study generated several key findings, including the need for tools and policies that establish a strong link between evaluation and decision-making, expanding the scope of engagement in data interpretation beyond primary users, and extending the dissemination of findings to external stakeholders. Implications for practice and research are discussed.

Developing Evaluation Capacity Building Competencies: Participant Reflections From the Evaluation Capacity Case Challenge

In 2023, McGill University’s Max Bell School of Public Policy hosted the inaugural Evaluation Capacity Case Challenge (EC3) competition with a cohort of 19 selected graduate students and early-career professionals studying or working in Canada or the United States. It was a multifaceted learning opportunity for participants to expand evaluation capacity building (ECB) competencies and served as a bridge between formal education and real-world practice. This practice note offers reflections from five students representing all teams and one coach on how EC3 supported competency development as outlined by the Canadian Evaluation Society and the American Evaluation Association. Focused on domains related to professional reflection, technical and methodological skills, situational context, as well as management and interpersonal skills, this article explores the role of EC3 in honing skills specific to ECB, preparing evaluators to excel in their roles and champion ECB in diverse and evolving contexts.

Learning From Evaluation Data: Discoveries From the Inaugural Evaluation Capacity Case Challenge

Evaluation competitions, a process that has existed for more than three decades, are a mechanism to promote program evaluation and enhance evaluation capacity. Integrating these competitions into evaluation education extends traditional teaching models beyond theory-dense learning contexts. This practice note delves into the Evaluation Case Competition Challenge (EC3), a case challenge designed to advance knowledge on evaluation capacity building (ECB). Insights from an end-of-experience survey are distilled into three discoveries: successful elements of the multifaceted competition structure and exploring the case challenge as a valuable opportunity for ECB, as well as a focus on student celebrations and suggestions. These discoveries provide insight into the learner experiences and recommendations for the future trajectory of EC3. Five guiding principles for envisioning evaluation learning competitions are offered for consideration.